Wiki: Collected feedback on interop testing, methods, living docs and specs

ActivityPub Testing
,
1

:point_right: This is a wiki post and anyone is encouraged to Edit and add additional sources.

ActivityPub Test Suite

Needs

Inclusion Anyone can write BDD scenarios, because they are written in plain English. Think of The Three Amigos.
Clarity Scenarios focus specifically on the expected behavior of the product under development, resulting in less ambiguity for what to develop.
Streamlining Requirements = acceptance criteria = test cases. Modular syntax expedites automation as well.
Shift-Left Test case definition inherently becomes part of grooming.
Artifacts Scenarios form a collection of test cases. Any tests not automated can be added to a known automation backlog.
Automation BDD frameworks make it easy to turn scenarios into automated tests.
Test­-Driven Most BDD frameworks can run scenarios to fail until the feature is implemented.
Code Reuse “Given-When­-Then” steps can be reused between scenarios.
Parameterization Steps can be parameterized. For example, a step to click a button can take in its ID.
Variation Using parameters, example tables make it easy to run the same scenario with different combinations of inputs.
Momentum Scenarios become easier and faster to write and automate as more step definitions are added.
Adaptability Scenarios are easy to rewrite as the products and features change.

  • Notes from @jfinkhaeuser:

    • There are a few traps here in the BDD summary that are worth picking apart in a project:
      1. Writing BDD scenarios is more of a challenge than the “Inclusion” point suggests. They may be plain English, but it’s easy to get them wrong.
      2. “The BDD community” is less focused on testing than on collaboration between different stakeholders. This has subtle results on tool usage.
      3. Step re-use is relatively low when scenarios are optimally written for inclusion & maintenance.
      4. Step parametrization can lead to less inclusion, as each step becomes more like a function.
      5. Typically, the solution to the above is a three layer archictecture (BDD, steps, functions representing interactions hiding implementation details) rather than the suggested two layers, BUT this implies the bottom layer is deeply technical and tightly coupled to the product.

  • Tests for an ActivityPub implementation checklist (project ports Go-Fed to Elixir)

  • Sytest: Black-box integration testing for Matrix homeservers (thanks @realaravinth)

    • Example of how Matrix does testing of Synapse server. Supports plugins for extension.

  • Part of W3C AP standardization process a large collection of User Stories was collected.

    • Can be used as input / inspiration to templates, best-practice for writing Gherkin scripts.
    • More user stories collected after official gathering process was closed.

  • @hazel toots about a C# Test Suite that is part of ActivityPubSharp library project.

  • @fr33domlover provided behavior pseudocode related to Vervis.

Click to expand and view the Vervis behavior scripts (text by fr33domlover)

Just to put this in the shared field, here’s an example of a quite-detailed behavior description I have for an Activity Handler that I just finished implementing (it’s very unusually long and complex, because the handler code itself is complex :P)

-- Meaning: An actor accepted something
-- Behavior:
--     * Check if I know the activity that's being Accepted:
--         * Is it an Invite to be a collaborator in me?
--             * Verify the Accept is by the Invite target
--         * Is it a Join to be a collaborator in me?
--             * Verify the Accept is authorized
--         * Is it an Invite to be a component of me?
--             * Nothing to check at this point
--         * Is it an Add to be a component of me?
--             * If the sender is the component:
--                 * Verify I haven't seen a component-Accept on this Add
--             * Otherwise, i.e. sender isn't the component:
--                 * Verify I've seen the component-Accept for this Add
--                 * Verify the new Accept is authorized
--         * If it's none of these, respond with error
--
--     * In collab mode, verify the Collab isn't enabled yet
--     * In component mode, verify the Component isn't enabled yet
--
--     * Insert the Accept to my inbox
--
--     * In collab mode, record the Accept and enable the Collab in DB
--     * In Invite-component mode,
--         * If sender is component, record the Accept and enable the Component
--           in DB
--         * Otherwise, nothing at this point
--     * In Add-component mode,
--         * If the sender is the component, record the Accept into the
--           Component record in DB
--         * Otherwise, i.e. sender isn't the component, record the Accept and
--           enable the Component in DB
--
--     * Forward the Accept to my followers
--
--     * Possibly send a Grant:
--         * For Invite-collab mode:
--             * Regular collaborator-Grant
--             * To: Accepter (i.e. Invite target)
--             * CC: Invite sender, Accepter's followers, my followers
--         * For Join-as-collab mode:
--             * Regular collaborator-Grant
--             * To: Join sender
--             * CC: Accept sender, Join sender's followers, my followers
--         * For Invite-component mode:
--             * Only if sender is the component
--             * delegator-Grant
--             * To: Component
--             * CC:
--                 - Component's followers
--                 - My followers
--         * For Add-component mode:
--             * Only if sender isn't the component
--             * delegator-Grant
--             * To: Component
--             * CC:
--                 - Component's followers
--                 - My followers
--                 - The Accept's sender

(It’s the Project actor handling an Accept activity)

Example of a a simpler description, for Project handling an Add activity:

-- Meaning: An actor is adding some object to some target
-- Behavior:
--     * Verify my components list is the target
--     * Verify the object is a component, find in DB/HTTP
--     * Verify it's not already an active component of mine
--     * Verify it's not already in a Add-Accept process waiting for project
--       collab to accept too
--     * Verify it's not already in an Invite-Accept process waiting for
--       component (or its collaborator) to accept too
--     * Insert the Add to my inbox
--     * Create a Component record in DB
--     * Forward the Add to my followers
projectAdd


  • @stevebate on fedidevs chat: What does spec compliance even mean?

    • (@aschrijver: Imho it is only meaningful in context of a Compliance Profile)
    • @stevebate: Howto compliance test “compliant, but unsupported Activity” (e.g. Travel)?
    • @trwnh: Test suite should test normative + “unstated logically implied” requirements.
    • @j12r: Made test requirements notes. make test-fediverse is needed.
    • @helge: Test suite must make bugs visible. Finds replies collection hardly implemented.
    • @stevebate: Useful tool probes a server to see what it supports via AP endpoint API.
    • @helge: Built something like that on proof.mymath.rocks. “Most stuff is optional”.
    • @trwnh: Explains how the underspec’ed replies collection works.
    • @j12r: Asks preferred techstacks, to align test suite technology (e.g. Docker-based).
    • @snarfed: Local + live testing ideal, but live preferred. See WebMention.rocks example.

  • @aschrijver: Found out (via HN) about the TestContainers project:

Fediverse Testing Practices

Needs

@aschrijver: Just sharing this for fun:

TigerBeetle uses a game to simulate distributed network communication (click to expand)

Tigerbeetle is a distributed DB made in Zig specifically for financial transactions and uses WebAssembly in interesting ways. The folks at the project have developed ‘a game’ that simulates actual network interactions and you can do can do all kinds of things to test the network. Here’s the demo. Who knows some day the Friendliverse in Mimic provides similar fun way of conducting tests :stuck_out_tongue_winking_eye:

Living Documentation

Needs

Living specifications

Needs

  • Key realization: AS/AP is a framework with which you can model any msg exchange.

    • Hence tackling major fedi challenge of substrate formation is a requirement for interop.
    • Dev community is slowly gaining this insight, recently in Next step for ActivityPub.
    • Compliance Profiles are a major feature that can be the basis for substrate formation.

  • Automation of manual procedures related to Fediverse Enhancement Proposals.

    • @helge PR’ed Python scripts and docs to help publishing FEP’s as web pages.

  • Statement was made (by @helge?) that W3C Verifiable Credentials Data Model contains best-practices as input for writing AP Extensions.

  • Various FEP’s are being outfitted with Behaviour Test scripts in Gherking format.

  • FEP Process discussion in Usage of PR’s for FEP’s on SocialHub:

    • Feedback is dispersed between Issue / PR / Forum topic discussions. Not good.
    • These tools not only don’t integrate well, they are non-inclusive, made for technical audience.
    • @by_caballero: Summarizes the problem statement neatly (read their post).

  • @trwnh in Fediverse Devs chatroom: Evan asked to rewrite Mastodon Move as a FEP, but most masto docs I wrote should be moved to FEP’s. They need additional “caveats and limitations” sections.

  • @runevision: Created great AP client comparison Google Sheet. Lists Fediverse features.

  • @aschrijver: GAIA-X in Federated Catalogues (unrelated to fedi) uses BPMN diagrams to model behavior.

  • @j12r: How do we keep fedi architecture consistent? Do we need an architecture board?

    • @hrefna: Current fedi encourages lack of interop, integration requires Θ(n^2) testing.
    • @hrefna: See this. Need a metamodel, describe extensions, core features, handshake.
    • @jenniferplusplus: We should publish suites of compatibility tests. No central authority.
    • @j12r: Used to run metamodel.com, see e.g. this article on terminology.
5 Likes