Fediverse Relays

I’m creating this topic for discussions about Fediverse relays: sharing technical notes, discussing implementations, documentation collaboration, and so on. The goal is to create an FEP to document the relay-related status quo to help other developers who need relay support (building a server that uses a relay, building a relay, etc.).

AFAIK, there are two relay protocols for the Fediverse, the Mastodon relay protocol and the LitePub/Pleroma protocol. I don’t know how much interop there is between them or if relays must do some normalization based on the type of relay client. I’m still investigating that topic.

I’ve been studying the Activity Relay (Go) implementation and @trwnh has been looking into pub-relay (Crystal). Are there other common relay implementations we should review? The relay protocols and behaviors will probably vary a bit depending on the implementation so it would be nice to have decent coverage of the variants.

I’ve started creating a small relay implementation in Python for educational and testing purposes.

/cc @bumblefudge

6 Likes

@stevebate have you also read the (WIP) fediverse auxiliary service Provider proposal from Mastodon? It’s like relays but a bit different

2 Likes

i’m concerned the FASP stuff seems to be moreso a bunch of bespoke APIs and less useful for providing an actual service (instead of an “auxiliary service”) over something like ActivityPub. specifically i was interested in defining a Relay Actor and how it should behave

5 Likes

warning: what follows is a draft of a draft


slug: “3416”
authors: a a@trwnh.com
status: DRAFT
dateReceived: 2024-04-03

FEP-3416: Relay Actors

Summary

This FEP aims to document different types of relay actors across the fediverse, as well as the mechanisms that power them. Current behavior of existing relay software can be very fragile; as a result, this FEP will primarily aim to define standard behaviors for how relay actors ought to operate, rather than detailing the intricate idiosyncracies of each existing implementation.

Introduction

(This section is non-normative.)

[Mastodon] added support for “federation relays” in 2018, where a “federation relay” was defined as an actor to which you can send public posts, and from which you receive other public posts. This was done to address the rise of “follow-bots”, which were unmonitored accounts that followed other accounts solely for the purpose of populating Mastodon’s “federated timeline” feature. [Pleroma] added support for “message relays” in 2018 as well, with largely similar logic, except that Mastodon asks for the inbox endpoint’s IRI, and Pleroma asks for the actor document’s IRI (and then derives the inbox from there).

Current implementations of the relay actor are as follows:

  • pub-relay
  • ActivityRelay
  • aode-relay
  • Pleroma; every instance of Pleroma can optionally serve as a relay for all local public posts, if the admin enables a particular setting

Current implementation notes

(This section is non-normative.)

pub-relay

JSON-LD terms

In general, pub-relay does not process JSON-LD on consumption. It expects the following terms verbatim, taken from the Security Vocabulary (https://web-payments.org/vocabs/security# at the time of implementation, later mostly superseded by https://w3id.org/security#):

  • Key
  • publicKeyPem
  • owner
  • publicKey
  • signature

It also normatively uses the following terms from the ActivityStreams context document:

  • endpoints
    • sharedInbox
  • inbox
  • type
  • preferredUsername
  • name

In other words, it implicitly expects compaction against the following context or equivalent (e.g. using the older Web Payments namespace):

{
	"@context": [
		"https://www.w3.org/ns/activitystreams",
		{
			"Key": "https://w3id.org/security#Key",
			"publicKeyPem": "https://w3id.org/security#publicKeyPem",
			"signature": "https://w3id.org/security#signature",
			"owner": {
				"@id": "https://w3id.org/security#owner",
				"@type": "@id"
			},
			"publicKey": {
				"@id": "https://w3id.org/security#publicKey",
				"@type": "@id"
			}
		}
	]
}

When producing activities, pub-relay will use the standard normative https://www.w3.org/ns/activitystreams context document, as well as the https://w3id.org/security/v1 context document in order to properly namespace the publicKey, owner, and publicKeyPem properties. (Key is unused during production of AS2 documents.)

HTTP Signatures

pub-relay requires and will verify HTTP Signatures according to draft-cavage-06. It checks for the following conditions:

  • The keyId is present
  • The signature parameter is present
  • An actor can be extracted from keyId, either by resolving it directly, or if it resolves to a Key, by resolving the owner

The actor of the activity will be ignored completely, in favor of the actor extracted from the HTTP Signature’s keyId. The resolved actor will be cached for 2 days.

The signature verification method supports the following key parameters:

  • SHA256 digests, base64-encoded
  • RSA public keys

Subscribing to the pub-relay actor

pub-relay considers the following activities valid for subscribing to the relay actor:

  • The type array includes Follow

Upon receiving a valid subscription activity,
pub-relay will check for the following conditions:

  • The inbox of the Follow actor (extracted from the HTTP Signature’s keyId, ignoring any explicitly defined actor of the activity) is a valid URI
  • The object.id exactly matches https://www.w3.org/ns/activitystreams#Public

Note that pub-relay will NOT handle follows for any other object, including for any of the following:

  • as:Public
  • Public
  • the relay actor’s id

Pub-relay will then save the subscription along with the actor’s domain, the actor’s inbox, the Follow activity’s id, and the actor’s id. An Accept activity will be generated by the relay actor and wrapping a reconstructed Follow activity, containing the actor.id, Follow.id, and with the object set to the relay actor. Finally, pub-relay will respond with an HTTP 202 Accepted, and the text “OK”.

Unsubscribing to the pub-relay actor

pub-relay considers the following activities valid for unsubscribing from the relay actor:

  • The type array includes Undo and the object is an embedded object whose type array includes Follow; alternatively, the type array includes Reject

Upon receiving a valid unsubscription activity, pub-relay will unsubscribe the actor’s domain. Finally, pub-relay will respond with an HTTP 202 Accepted, and the text “OK”.

Forwarding rebroadcastable activities

pub-relay considers the following activities valid for rebroadcast (inbox forwarding):

  • The type arrays do not match any of the following patterns:
    • Follow
    • Undo Follow or Reject
    • Accept
  • The activity was published within the last 30 minutes (otherwise, pub-relay will skip the “old” activity with an HTTP 200)
  • The activity does not have an id that was already previously processed (otherwise, pub-relay will skip the “already seen” activity with an HTTP 200)
  • to or cc includes https://www.w3.org/ns/activitystreams (pub-relay will not recognize as:Public or Public)
  • The type array includes any of the following:
    • Update
    • Delete
    • Undo
    • Move
    • Like
    • Add
    • Remove
    • Create (if signature is present)
    • Announce (if signature is present)

Upon receiving an activity that is valid for rebroadcast, pub-relay will deliver the request body as-is to all relay subscribers. Finally, pub-relay will respond with an HTTP 202 Accepted, and the text “OK”.

Relaying relayable objects

pub-relay considers the following activities valid for relay (Announcing an object):

  • The type arrays do not match any of the following patterns:
    • Follow
    • Undo Follow or Reject
    • Accept
  • The activity was published within the last 30 minutes (otherwise, pub-relay will skip the “old” activity with an HTTP 200)
  • The activity does not have an id that was already previously processed (otherwise, pub-relay will skip the “already seen” activity with an HTTP 200)
  • The activity is not valid for rebroadcast
  • The type array includes any of the following:
    • Create
    • Announce

Upon receiving an activity that is valid for relay, pub-relay will construct an Announce activity wrapping the activity’s object.id. Finally, pub-relay will respond with an HTTP 202 Accepted, and the text “OK”.

ActivityRelay

aode-relay

Pleroma

Standardization

Subscribing to a relay

Subscribing to a relay is done with the standard Follow → Accept Follow flow described in [ActivityPub]. Use of https://www.w3.org/ns/activitystreams#Public as the object of a Follow SHOULD be deprecated. The actor of the activity SHOULD be used instead of extracting the actor from the HTTP Signature, subject to a same-origin policy check.

Extending the relay model

Alternative approaches

(This section is non-normative.)

[FEP-1b12] works similarly in that it wraps received messages in Announce, but differs from the classical relay actor in certain ways:

  • imposing additional restrictions, such as mandating a type of Group and the use of inReplyTo
  • leaving moderation as optional instead of required
  • wrapping the activities with an Announce as-is, rather than Announcing the object of the Create/Announce

Guppe also works similarly in that it Announces anything sent to it, except that it operates at a user level instead of a service level.

References

Copyright

CC0 1.0 Universal (CC0 1.0) Public Domain Dedication

To the extent possible under law, the authors of this Fediverse Enhancement Proposal have waived all copyright and related or neighboring rights to this work.

4 Likes

@trwnh I hear you, I have been giving a lot of feedback to try to nudge it closer to AP

2 Likes

Here’s a blog post I wrote almost a month ago trying to explain (Mastodon-flavored) relays. It’s in the queue for the work blog, but in the meantime I’m posting it here in case it’s useful to anyone.

Notes on talking to Fediverse relays

by Darius Kazemi
Sep 10 2024

I was recently trying to write some software that can subscribe to a Fediverse/ActivityPub relay, and I was surprised to find that there’s very little documentation about relays online. Usually my go-to for niche ActivityPub implementation stuff is the SocialHub Fediverse Developer Portal, but I couldn’t find much there. I ended up having to dive into a bunch of source code and puzzle some things out from various READMEs and don’t want anyone to have to do that again!

But before I get into the tech, let’s talk a bit about relays and why we might want to talk to one.

What is a Fediverse relay?

A relay is a piece of software designed to solve a Fediverse-specific problem, so I’ll describe the problem first.

The problem: it’s hard to crawl a network

Let’s say I’m starting a new Mastodon server. (I’m using Mastodon here but the general principles apply across other software too.) I put this new server online at coolsite.example. That server doesn’t actually have any way of knowing that mastodon.social or any of the ~20,000 other servers out there exist. Normally the way a server learns about other servers is through user action. In the case of coolsite.example it just came online, and as such I am the only user with an account at @darius@coolsite.example. Just lil’ ol’ me on an empty server with no messages to read.

That is, until I type @elonjet@mastodon.social into my search bar. What happens then is (roughly and simplified):

  • coolsite.example contacts mastodon.social and asks “is there an account on your server called elonjet?”
  • mastodon.social sends a message back to coolsite.example saying “yes there is, and here is where to find it”
  • coolsite.example shows me the profile for elonjet
  • If I press “Follow”, it sends a follow request to the inbox for @elonjet@mastodon.social, which automatically sends back an approval. The approval is a promise from mastodon.social that it has put @darius@coolsite.example into a list of accounts that should be sent a copy of messages that @elonjet@mastodon.social posts
    • Being on this list makes my account a “follower”! From now on, when @elonjet@mastodon.social makes a post, it will send a copy of the post to coolsite.example, which will distribute it to any of its users who follow @elonjet@mastodon.social
    • It’s important that only one copy is sent to coolsite.example, because if there were 100 followers on that server, you wouldn’t want to send 100 copies of the same message to the server. That could add up quickly, so instead coolsite.example keeps track of followers on its end and puts the message into all of their home feeds

There are other rules around boosts and locked accounts and all that but I’m trying to keep things simple so… maybe a post for another time.

The thing is, at this point there is one user on the server (me) following one user on another server (Elonjet). This is coolsite.example’s entire model of the Fediverse. So when I am curious about finding new people to follow and I say “show me every post with #kittens”, it shows me posts that use #kittens from me, and #kittens posts from Elonjet, and from other people’s posts that Elonjet has boosted with #kittens, and… that’s all. It’s likely that search results are empty even though there are probably lots of #kittens posts out there.

The “normal” way to grow a server’s model of the Fediverse and get more posts into it is for all the users on a given server to follow whoever it is they naturally follow, and then the server builds up a database of known accounts. Because of boosting, this isn’t even necessarily just accounts the user on a server follows. Over time, and with enough people doing enough following, a server eventually does learn about all sorts of corners of the network.

But if you are on a server of one person the model breaks down. It can be difficult for small servers to learn about the network.

A solution: relays

A relay is a piece of software designed to solve the above problem. A relay is basically a special kind of Fediverse server with a single “account” (though it’s a phantom inbox, not really an account, for complicated reasons). The way it works is it basically:

  • I decide I want more content to browse, so I google “Fediverse relays” and find a list somewhere

  • When I see a relay that looks active, I (as the admin of my server) subscribe to the relay via a special subscription page in Mastodon admin interface

  • Subscribing to the relay means we enter an agreement with the relay. What we agree to is: I will send the relay every public post made by my users, and in return it will send my server every public post made by every other server that subscribes to the relay.

:bulb: Note that this is about servers, not individual users. So if I hook my server of 1 user up to a relay and there is another subscribed server on there with 10,000 users, then that server now gets the posts from my 1 user, and my server gets the posts from 10,000 users. Most of those posts won’t end up in my home feed because I don’t follow any of them, but the posts will now live in a database so when I search for #kittens I will get any #kittens posts from those 10,000 users! Think of it as a commons of public content that everyone is going in on. You give content in order to get content.

Also important: there’s no guarantee that a relay will give you good content. Most public relays are hardly moderated at all, so it’s possible that you will hook up to a relay and suddenly see a bunch of terrible garbage in your feeds. As admin, you can always unfollow the relay on behalf of your server, and block/defederate it if it isn’t honoring the unfollow.

How to write software that subscribes to a relay

I have known about relays for many years but have never tried to write software that subscribes to a relay. I wanted to try it out and I thought there would be technical documentation somewhere but there wasn’t any I could find! I guess this makes sense, as relays aren’t standardized, they are just a tool that a bunch of people decided was a good idea and started making and using.

So I had to read a bunch of source code and do a lot of experiments to figure out the subscription step.

:bulb: Warning! There are two major relay subscription protocols (I told you it wasn’t standardized!). Broadly speaking there is a Mastodon-compatible protocol and a Pleroma-compatible protocol. Most relays support both but it’s important to know that there are two different methods. In this article I discuss the Mastodon-compatible method because that’s the flavor of ActivityPub I am most experienced with.

If you want to follow a relay, the first thing you need to know is the location of its inbox. This will be written on the website of a given relay and will be something like https://relay.example/inbox. (To further complicate things, some relays are private and will require you to email someone first to pre-approve you for this. But there are also public relays that accept anyone.) What you want your server to do is send a bare-bones ActivityPub Follow event to that inbox. That is, you will HTTP POST it like you’d POST a follow request to any normal ActivityPub inbox. The main difference is the content of the object field:

{ 
  '@context': 'https://www.w3.org/ns/activitystreams',
  id: 'some-long-unique-id-string-here',
  type: 'Follow',
  actor: 'https://coolsite.example/actor',
  object: 'https://www.w3.org/ns/activitystreams#Public'
}

To get really specific here:

  • @context is a string that says this is ActivityStreams (and therefore ActivityPub) data
  • id is a unique ID (unique to the server of origin — other servers likely use a combination of “domain + id" to disambiguate messages in case two servers happen use the same ID for messages)
  • type lets everyone know this message is an Activity Vocabulary Follow Activity
  • actor is a URL that resolves to an actor object that, importantly, has a property on it called endpoints which has a property on it called sharedInbox . The sharedInbox value is going to be the URL where you would like the relay to POST messages to your server. For a real-world example take a look at https://mastodon.social/actor, where the value of sharedInbox is (at the time of writing) "[https://mastodon.social/inbox](https://mastodon.social/inbox)" . The actor should also specify an inbox property. The sharedInbox and inbox can be the same, or different, it’s up to you how you want to handle messages.
    • For the purposes of this blog post let’s say my inbox is set to "https://coolsite.example/inbox" and my sharedInbox is "https://coolsite.example/inbox/shared"
  • object MUST be the string "https://www.w3.org/ns/activitystreams#Public"

So the actor and the object are the bits that are different from a normal follow request.

If you POST the object to the correct inbox, and the relay is a public one that doesn’t require you to contact an admin to get access to (or you have been pre-approved for a private relay), you should within a few seconds or minutes receive back to the inbox of the actor an Activity Vocabulary Accept Activity with the object you just posted to it embedded inside. In my made-up example, we can expect the following message to be POSTed to "https://coolsite.example/inbox":

{
  '@context': 'https://www.w3.org/ns/activitystreams',
  id: 'another-long-unique-id',
  actor: 'https://relay.example/actor',
  type: 'Accept',
	object: { 
	  '@context': 'https://www.w3.org/ns/activitystreams',
	  id: 'some-long-unique-id-string-here',
	  type: 'Follow',
	  actor: 'https://coolsite.example/actor',
	  object: 'https://www.w3.org/ns/activitystreams#Public'
	},
  to: [ 'https://coolsite.example/inbox' ]
}

This lets you know that the relay is going to start sending you stuff! You should log this somewhere so you can unsubscribe later if you need to (by sending an Undo/Follow). If you messed up somewhere, you might get a similar message but with a type of 'Reject'.

Pretty soon you should start seeing random ActivityPub messages POSTed to whatever you said your sharedInbox is (in my example, "https://coolsite.example/inbox/shared"). And that’s it: you’re connected and getting messages from accounts and servers on the Fediverse you didn’t even necessarily know existed. Do with those messages whatever you will: show them in a “federated feed” view, index them for local search if they have flagged that they want to be indexed, etc etc.

5 Likes

Could you share links to their source code?

I assume the first one is yukimochi/Activity-Relay, but I’ve never heard about pub-relay (found a similar project on Github but it seems to be abandoned long time ago).

You may want to review pleroma/ActivityRelay, a mature relay implementation written in Python, and buzzrelay, a different kind of relay written in Rust, which powers https://relay.fedi.buzz service.

1 Like

That’s the one: GitHub - noellabo/pub-relay: A service-type ActivityPub actor that will re-broadcast anything sent to it to anyone who subscribes to it.

It was I think originally written and hosted on source.joinmastodon.org but that hasn’t been a thing for years.

1 Like

Yes, that’s it. I had linked to it in the thread from which this one branched. I should have linked it again in this thread.

1 Like

It looks like there some good material here about relays. It mostly overlaps with what I have. One aspect that’s not covered (unless I missed it), is the JSON LD Signatures required for many of the messages passing through the relay. [Mastodon documentation]

We might also need to agree on some core terminology. It looks like pub-relay defines message forwarding and message relaying and I don’t fully understand the significance of the distinction at this point. Other relay implementations simply “relay” messages. I also would like to avoid writing this in terms of Mastodon (or Pleroma) specifically, but I don’t have a good generic name for a server that publishes to and consumes from a relay server.

Any ideas about how to proceed? Do we merge the material? The blog article (which looks good) is typically a different type of document than an FEP (less conversational). It could be useful to have an informational FEP and one or more related blog articles.

1 Like

Part of the goal of the FEP I was writing (oh god that was back in APRIL???) was to try and boil these different terms and mechanism down into something that could be standardized.

  • pub-relay “forwarding” will use inbox forwarding to POST the activity directly to all followers, unmodified.
  • pub-relay “relaying” will Announce the object of any Create/Announce that matches the criteria listed. So in Mastodon terms, it will “boost the post”.

Yeah I hadn’t gotten around to writing that section yet. Also I think there was maybe some reason I didn’t get around to writing it down? I don’t remember if I got around to looking at the LDSig stuff in pub-relay or if the LDSig requirement actually exists only on the Mastodon receiving side.

In theory, any actor can deliver to a relay. The thing I wanted to define in the FEP was what I would now/currently call a “client-to-client protocol”. I don’t want to hardcode assumptions that there is a server/instance sending or receiving any activities against the relay.

Probably we can proceed by continuing to trawl through the source code of various relay implementations and see if any of them do anything unique or weird. Then we document that in the FEP. Then we can start developing a more modern “protocol” for how an actor of type https://w3id.org/fep/3416/Relay is required or recommended to behave. With test cases and algorithms. Then finally we can push the FEP and try to get implementers on-board with at least recognizing these actors and their codified behaviors.

2 Likes

Interesting. This seems significantly different than what Activity-Relay does (unless I just haven’t seen it yet). AFAICT, it doesn’t do any relay-initiated announcements. Are those boosts are displayed In Mastodon as being boosted by the relay itself (since that’s the signer/actor)?

To me, that’s surprising behavior for a relay. However, given a core relay “protocol”, other types of behavior can obviously be supported beyond just relaying messages. For example, a labeler could analyze posts and add hash tags automatically, or it could monitor messages for illegal content and notify moderators, or whatever. However, I consider these to be valued-added behavior, not relaying pe se. I’ll need to think about it more, but I think of a relay as relaying messages unchanged. Forums and group servers like Guppe are other examples of functionality that is relay-ish, but something more. In any case, I’m sure we’re going to see lots of these “creative” variations on the relay-related theme.

I agree that it makes sense to talk about an actor publishing or consuming from a relay (vs. a server), but I’m still don’t know a good name for that type of actor. In the relay, it makes sense to me to refer to “relay actors”. Most relays have a single relay actor, but some (like buzzrelay) can have many relay actors. I don’t think it changes the “protocol” though.

Speaking of buzzrelay, does it currently only relay messages from actors with a follow relationship (e.g., a Mastodon instance that has configured buzzrelay as a relay)? I thought I remembered it using streaming federation timelines before Mastodon required authentication to use them.

1 Like

I don’t think Mastodon generates a “boost notification” for the user when their Public post is sent to a relay, no. This is mostly due to the recipient of a relayed object being the followers of the relay, and in this case it would be the Mastodon “instance actor” that receives the Announce. I think the Announce isn’t sent to the author of the post, and generally users aren’t allowed to follow relays (only admins have the power to make their instance follow relays from the admin panel).

I’d like to at least give some consideration to how relays are to be moderated. I don’t think a ton of extra functionality makes sense per se, but I haven’t identified those boundaries yet between what makes sense for a Relay actor and what doesn’t.

In this case I was intending to separately write up FEP-9988 “Federated Forums” to define a “forum protocol”, again with test cases and algorithms. There’s definitely a lot more that a 9988/Forum actor would do that a 3416/Relay actor would not do. But this is also something that the SocialCG Forum TF is incubating right now, so there’s room for things to develop on that front (and certainly room for cross-pollination!)

One thing at a time, though…

I agree, but I was thinking about what happens when Mastodon receives an Announce that was generated by a relay actor. I’m wondering if there’s code in Mastodon to treat that as a special case (since any actor can send a ld-signed message to a Mastodon inbox AFAIK. I haven’t (yet) seen behavior specific to receiving such a message from a relay (like unwrapping an Announce).

I may need to spin up a pub-relay server to see if my Mastodon instance does anything special with the relay-wrapped activity. I’m curious now.

1 Like

In my yet-to-be-released Fediverse Schema Observatory I have picked up a few thousand Announce activities from 3 different servers running a piece of software identifying itself as activity-relay from the yukimochi github repo.

image

Presumably it is related to this function: Activity-Relay/api/resolver.go at e602188efd3100d36cedf5c218e27c00f0e22417 · yukimochi/Activity-Relay · GitHub

Do with that info what you will!

1 Like

The yukimochi activity-relay supports both the Mastodon relay protocol and the LitePub/Pleroma protocol. I haven’t dug deeply into the LitePub protocol yet, but it looks like the originating servers use Announce activities to publish activities for relaying (based on the source code). I’m not sure why at this point.

I think FediBuzz gets content from instances that donate their API tokens: #FediBuzz: Donate a token

Because everyone can read announced posts. Relays that use custom protocols exclude regular users (non-admins), as well as majority of Fediverse implementations.

1 Like

Has FediBuzz seriously not stopped this horrific security practice? I’m honestly tempted to suggest Mastodon block their user agent from accessing the streaming server instead of properly implementing as a relay. They’ve been told about how horrifically bad this practice is for months, if not a year.

Anyway, we’re working to move Mastodon to short-lived access tokens and refresh tokens for better security, so this practice will stop working.

What’s bad about it? It gives read access to their entire server, gives access to followers-only and mentioned-only posts, not just public posts.

1 Like

I don’t think it’s “excluding” anyone or anything? The alternative to using an Announce is to just POST the activity unmodified. There’s no good reason to wrap it in an Announce except that it trips up HTTP Signature checks when the actor of the payload activity is not the entity delivering the activity, or not on the same web origin/domain. This is the fault of brittle implementations that overly depend on HTTP Signatures even in ways that don’t make sense.

Are there any blocking-related considerations to wrapping it in an Announce (either in principle, or with current implementations and their approaches to Authorized Fetch)?

[This isn’t meant as a gotcha question, I genuinely have no idea!]

1 Like