I’ve heard there’s still trouble getting the official test suite up and running in its original form. I don’t want to detract from that effort, if it is still going on.
However, I did want to have a bit more of an automated experience. And, in a great exercise of NIH syndrome, I started hacking on go-fed/testsuite, built on go-fed: https://github.com/go-fed/testsuite
When a user wants to begin testing their AP application, the server launches a TestRunner, which entirely contains its own hermetic environment: an isolated in-memory DB, isolated web serving paths, isolated AP actors. Yes, it basically spins up a whole S2S+C2S “federated app” that acts as a giant shim for the tests to leverage. Yay for the go-fed library!
Here’s a sample screenshot of the very bare-bones UI showing that a test needs further input and is awaiting the user to input more. There’s a table of pending tests (blocked by the instructions or simply being late in the order) and a table of completed tests:
If a test succeeds or fails, each test has a very detailed log of what it was doing to execute those tests, so end-users can get an idea of what was going on and where things started going wrong:
Clearly there’s more work to be done (more tests to be written, a hell of a better UI to make). I would definitely appreciate help if folks are interested.
It is my hope that most of these tests are automated and there are as few “on your honor, check the box” type of questions as possible.
All of this was hugely inspired by the original official test suite, thank you to those who built it (@cwebber I believe!).