Unofficial Test Suite (go-fed/testsuite)

Software Go-Fed
1

Hi all,

I’ve heard there’s still trouble getting the official test suite up and running in its original form. I don’t want to detract from that effort, if it is still going on.

However, I did want to have a bit more of an automated experience. And, in a great exercise of NIH syndrome, I started hacking on go-fed/testsuite, built on go-fed: https://github.com/go-fed/testsuite

When a user wants to begin testing their AP application, the server launches a TestRunner, which entirely contains its own hermetic environment: an isolated in-memory DB, isolated web serving paths, isolated AP actors. Yes, it basically spins up a whole S2S+C2S “federated app” that acts as a giant shim for the tests to leverage. Yay for the go-fed library!

Here’s a sample screenshot of the very bare-bones UI showing that a test needs further input and is awaiting the user to input more. There’s a table of pending tests (blocked by the instructions or simply being late in the order) and a table of completed tests:

Screen Shot 2020-07-15 at 21.39.31
Screen Shot 2020-07-15 at 21.39.311433×814 60.1 KB

If a test succeeds or fails, each test has a very detailed log of what it was doing to execute those tests, so end-users can get an idea of what was going on and where things started going wrong:

Screen Shot 2020-07-15 at 21.41.54
Screen Shot 2020-07-15 at 21.41.541368×624 46.9 KB

Clearly there’s more work to be done (more tests to be written, a hell of a better UI to make). I would definitely appreciate help if folks are interested.

It is my hope that most of these tests are automated and there are as few “on your honor, check the box” type of questions as possible.

All of this was hugely inspired by the original official test suite, thank you to those who built it (@cwebber I believe!).

2 Likes
2

FYI, interested parties can follow along the work being tracked in this mega issue

3

I’m definitely going to be taking a look at this! Thanks for sharing!

4

Stupid question: how do I run this? I’ve never dealt with Go. And, should I run this on a server that has a public IP or this isn’t needed?

5

Sneak peek: https://test.activitypub.dev/

Please feel free to file issues.

Caveats:

  • UI is ugly
  • The “C2S” option in the “new test” page does nothing. You’ll probably want to check the “S2S” one.
  • Once you begin the test, you’ll have to refresh the page manually to get updates. Once instructions show, testing is paused and awaiting your input
  • Did I mention the UI is ugly? I haven’t done but the bare minimum for the instructions, etc.
  • Known problem with the “done” button for the one test that has it.
  • Early on a test asks you to “Block” an actor, but later tests re-use that actor (oops)
  • One or two tests have a 5 second sleep. Hooray

To build and run it on your own:

$ go get github.com/go-fed/testsuite
$ go install github.com/go-fed/testsuite
$ ./$GOPATH/bin/testsuite -cert <your TLS pub key file> -key <your TLS private key file> -host $hostname

Not sure if it will work with self-signed certificates / localhost, as I didn’t consider testing locally. However, one of the test cases to port is “dont send HTTP requests to localhost” which will make this not a good way to test anyway.

2 Likes
6

I’ve updated the test suite at https://test.activitypub.dev to be much more pleasant-looking and hopefully more comprehensible when running the tests. Please take a look and file bugs against the test suite if needed.

2 Likes
7

Hi @cjs. The test-suite is currently down, I noticed.

8

Just wanted to mention here, should be good now. Thanks for pinging me on matrix. :slight_smile:

2 Likes
9

Looks like the cert expired again?

2 Likes
10

Pinging @cjs & @cwebber , for cert renewal on https://test.activitypub.dev/ not sure who owns the domain