I’ve been doing experimentation with ActivityPub compliance testing, but it might help to describe your specific objectives for the testing. There are several potential approaches to AP server testing. (I’m assuming you want server-side testing rather than client testing.)
- Interactive questionnaire
- Partly-automated (web) application
- Fully-automated regression test suite
(EDIT: Sometimes when someone asks for an ActivityPub test suite, they are really asking for a Mastodon microblogging interop test suite, which is quite different. This post is specifically about AP compliance testing.)
The original activitypub.rocks server test suite was a questionnaire that asked developers if they implemented (or believed they implemented) specific features. I wouldn’t really call it a test suite. It is more of an interactive checklist application.
The Go-Fed test suite has some automation, but it is still partly interactive. It’s designed to be hosted on a secure public website (versus non-SSL LAN operation). In that sense, it is similar to the webmention test suite. However, I found it difficult to use for local development purposes. Every time the test suite is run, it requires manual entry of information about the tested server. Individual tests cannot be run or re-run. The web application requires manual browser reloads to see interactive questions for the test steps. These issues could be resolved to some extent with additional development work.
For my purposes, I’ve been experimenting with creating a fully automated regression test suite. The idea is that the tests have no implementation details in them and can be re-used with different server implementations. They interact with a server implementation using a server-specific “driver” for introspection and server control. Creating the driver is non-trivial, but test automation is a benefit that continues to pay dividends during the ongoing development of an AP server.
Since many parts of the AP specification are optional, my test suite uses a configuration that will skip tests for functionality that the server doesn’t support.
The test suite is written in Python (pytest), but my goal is that it can support testing servers not written in Python. I’ve written drivers for Vocata and Bovine, which are implemented in Python. I’ve also written a driver for ActivityPub Express (Javascript, Node.js) and snac2 (C, partial support at the moment).
For the non-Python servers, the tests launch the server in an external process and control it remotely. This works well for ActivityPub Express and snac2 because they start quickly. It wouldn’t work so well with more complex server architectures like Mastodon, kbin, etc. In my testsuite, the existing tests are isolated (start with a fresh server instance for each test), but it might be possible to do some testing of larger servers without a reset between tests (needs experimentation).