Issue with Mastodon pulling posts & followers from new ActivityPub integration

jacliu · August 21, 2024, 3:51am

I am in the process of building an ActivityPub integration for Mirlo, an open source platform where users can publish their music and also maintain a blog on their artist page.

How I heard of SocialHub was from this Github issue inviting anybody from that community to join this forum to discuss incorporating the ActivityPub standard.

Note: I am a new user so I can only put 2 links in each forum post. The location of my dev instance is goaify[.]me which you can append to the below routes to view the JSON or webpage I am referring to.

So far I have built an Express.js / Node service which dynamically generates the Webfinger goaify.me/.well-known/webfinger?resource=acct:test1234@goaify.me and actor JSON-LD (goaify.me/activitypub/test1234 when the GET request is sent with Accept: application/activity+json header) based on the application’s API (e.g. api.goaify.me/v1/artists/test1234)

This makes the URL, url slug, artist name, artist ID, avatar, banner, bio, and links of the artist discoverable on Mastodon when searching for the ActivityPub user by their account name, in the above example, test1234@goaify.me. The HTML version of the artist page is located at goaify.me/test1234.

Basic Account Data
On a Mastodon instance, the account appears as shown below. It looks pretty good so far, but I wonder how often Mastodon refreshes its cache when the details change at the remote server?

Followers
At this stage of the implementation at goaify.me/activitypub/test1234/followers, I have hard-coded two followers for each actor (my personal Mastodon account and the actor itself). The follower count of 2 is detected on Mastodon (on instances which haven’t cached the previous follower count of 1), but the actual follower list is invisible with the message This user has chosen to not make this information available – why?

In the actor file I have already set discoverable, indexable to true and hide_collections to false.

Posts

The most challenging part is getting the blog posts located at goaify.me/test1234/posts to populate in Mastodon. I understand that Mastodon does not backfill posts from before an ActivityPub account was first looked up on that Mastodon instance, but I am not seeing it populate new posts either.

The error is Older posts from other servers are not displayed. Browse more on the original profile.

In the ActivityPub user’s outbox at goaify.me/activitypub/test1234/outbox, I have the blog posts and their contests listed in JSON format. I believe I read somewhere that Mastodon does not pull posts from the outbox, but rather requires them to be POSTed to an account following the remote user locally on that Mastodon instance.

It looks like I would need to first fix the followers flow so that it is not hard-coded, but actually accepts follow requests from ActivityPub servers and persists the list of followers somewhere (e.g. a database). For that activity I would also need to verify the HTTP signature being sent from the remote server on behalf of the user who is following?

Is it required to make each individual post accessible in JSON format at their own permalink URL in order for Mastodon to ingest it, or is an orderedItems list similar to what I have done in the outbox sufficient?

At what route should the orderedItems list of blog posts reside?

Any help from folks who have experience building a custom ActivityPub client, and knowledge of Mastodon’s behavior would be appreciated. I have tried to scour the documentation and forums as much as possible, but the docs seem a bit sparse and each ActivityPub implementation does not 100% conform to the spec.

silverpill · August 21, 2024, 8:55pm

I don’t know much about Mastodon, but I noticed that Create activities in your outbox and their objects have different owners:

Create.actor is https://goaify.me/activitypub/test1234
Note.attributedTo is https://goaify.me/test1234

Note should be attributed to activity actor.

This is required by ActivityPub standard where IDs must be “publicly dereferencable URIs”.

nightpool · August 21, 2024, 10:37pm

Correct. This is what it means for Mastodon to " not backfill posts from before an ActivityPub account was first looked up on that Mastodon instance". It will only know about the posts YOU send it. It will not look on your server for new posts.

As you might be able to tell from the name, ActivityPub is a pubsub protocol—it’s a way to publish new activities to your followers. The issue is in the title of your thread itself—Mastodon will never “pull” a post it hasn’t been notified about. You have to tell it about the post yourself before it will process it.

Correct. Before Mastodon can show your posts, you have to be able to accept followers from Mastodon instances. Only once you’ve accepted their follow request can you then proceed to send them your posts.

This is required by spec, but strictly speaking it’s not required for Mastodon to start showing your posts. 80% of the time, all you need is the POST. However, there are some rare cases where Mastodon will try to look up posts that you don’t sent it, like if someone on server A replies to your post, their followers on server B will try to look up the post they replied to. So it’s recommended, and it certainly helps with debugging, but it’s probably not the first thing you need to build.

Your call! ActivityPub does not specify the structure / layout of URLs, you can feel free to compose them however you like. However, in this case, orderedItems is completely useless for Mastodon federation. It’s good to implement eventually, but it’s completely barking up the wrong tree when it comes to getting Mastodon (or most implementations) to display your post.

So, to recap, your priority list as a new AP implementor is the following:

Actor endpoints — seems like you’ve got these working!
Webfinger — Mastodon needs this to display your actor, so it seems like this is working also
Receiving Follow requests and validating HTTP signatures — sounds like this is your next step
Sending your own Accept { Follow } messages, and generating HTTP signatures
Sending your own Create { Note } activities — this is the point at which Mastodon will display your posts.

LMK if that makes sense or if you need anything clarified!

jacliu · August 26, 2024, 5:12pm

Would like to thank all for the detailed input, particularly @nightpool.
It makes sense to proceed along the thread of getting “follows” working, then customize the application to POST new blog posts to the followers on Mastodon.

I have written the Express code to be able to parse the POST request which Mastodon sends upon a follow request, and extract both the relevant HTTP headers and request body:

HTTP Headers

headersToSign

(request-target): post /activitypub/test1234/inbox host: goaify.me date: Mon, 26 Aug 2024 17:04:14 GMT digest: SHA-256=byoTC4x/rKWnhg9+4XZeUDeesNVK17bbCSEdMm178F4= content-type: application/activity+json

signature

HQZW138MltJT/gSeUWj7PYQvY4xtBas9y1NxZ+p1Vph8UBr4LJOajwWfdc28bUiMOVYq9cWbQ29qy5zuLvAwujicbqnkX4s/iBb7m6sg2WFOMc/STv0/KncRQ8SqXV+2ZW64bZbVNgpy0CdPpjNhJO5oNAD5hFybTHx5IBhqm3TJttY5JyZEB4+R5vT1Q6L9fL4VaxjHv0RkA7fPM7PpbsrydibIeOZmHf7QtrL+5qWH+7C8eqKpmexEZ6j80/63KuH9zNrWe3tUastjOPPDO0s6LZOZnvJWMCkOQ1Pf7Tr772ZvEG2SDPDZXFwI4MW21s79PAqALTgGq5JLCeEP3Q==

HTTP Message Body

id https://mastodon.social/97ea7544-7d82-4ddf-9143-bc78bfa45ffe
type Follow
actor https://mastodon.social/users/jackie_liu
object https://goaify.me/activitypub/test1234

I also use Axios library and the Express JSON function to make a GET request to https://mastodon.social/users/mastodon_user#main-key to retrieve the publicKeyPem of the follower from the Mastodon server, and store it in a constant called publicKey.

publicKey

-----BEGIN PUBLIC KEY----- MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwyhIT8T0AftrbDhvil7L cCgzt0SVnUuONuk3lPTqwUCEJZYXIF88WLa6xFwQ2Pd19jT1kk3GbqDF86Kfy63t XFj8pTrgICvocaX5stLa64CU/L/K3fzpOktxYk53h2ujIfNm0V7AAF1y3ajQ/OIe LHHLOyXrrZxvPSEZWJigNK+bxNhszE09SnAyy39WlYuVpWRJQOvJjM7cusiHeqz+ 01ZBc9wTjZwgv0Vg924WJZqpuxvk8gg4Ecc/OWHIkpgEf3cieolJYb3sHbwo/HdH YR/NzWxzj7NE+AFk74qlJ4wLEqJfF5fKZCCBMTPYVwv0uKsJJVmPklHLnkWQQtRR NQIDAQAB -----END PUBLIC KEY-----

The headersToSign contains the parameters host, date, digest, content-type (outlined above) contained in the POST request that Mastodon makes to my ActivityPub instance as soon as the Follow request is made.

Edit: I’m able to verify the HTTP signature from Mastodon by converting the base64 encoded signature to a Buffer using the Buffer.from function and get a Signature verification succeeded in the console.

Here is the relevant code excerpt:

const crypto = require('crypto');

const remotePublicKeyPem = response.data.publicKey.publicKeyPem; 

    const publicKey = crypto.createPublicKey({
      key: remotePublicKeyPem,
      format: 'pem'
    });

    const verifier = crypto.createVerify('RSA-SHA256');
    verifier.update(headersToSign);
    verifier.end();

    const signatureBuffer = Buffer.from(signature, 'base64');

    if (publicKey !== undefined && signature !== undefined) {
      const isValid = verifier.verify(publicKey, signatureBuffer);

      if (!isValid) {
        console.log('Signature verification failed');
        return {
          error: "Signature verification failed"
        }
      }
      console.log('Signature verification succeeded');
    }

Next Steps

Next, I plan on writing the code which will allow the local ActivityPub account being followed to sign an Accept message using its own public key, and return that response to the Mastodon server. As well as update my /activitypub/test1234/followers route to read from a list of followers in a local database.

jacliu · September 2, 2024, 5:32pm

Does anybody happen to have an example (headers & request body) of the PUT request which needs to be sent when an ActivityPub server accepts a follow request from another ActivityPub server? I am guessing it needs to be signed in the same format as the follow request with a base64 encoded signature. What other fields are mandatory?

devnull · September 8, 2024, 11:29am

Sure, look up how it’s handled here: NodeBB/src/activitypub/inbox.js at 652d6c6e2b444156cf97febebc0c4fbbbed456b7 · NodeBB/NodeBB · GitHub

Note that the id of the activity needs to be unique and saved to your db. Some software will see send that id back in the accept, and only accept Undo(Follow) with that same ID originally received.