Can you clarify this point? Isn’t all public content typically readable by everyone?
Aren’t all Fediverse relay protocols “custom”, unless you consider Mastodon’s protocol a de facto standard given its prominence in the Fediverse? The Mastodon relay protocol doesn’t exclude regular users, but the Mastodon implementation only allows admins to configure relays.
Not any that I can think of. Blocking doesn’t make a difference in either case; the only time it could matter is if the object is not embedded and is only referenced by its id, at which point you would have to fetch it from its origin, and there might be access control involved with that. I guess if you Announce the post directly then it’s the same in both cases, but if you’re comparing an Announce to a forwarded Create then the properties of the Create are generally present (“leaked?”) since the activity is forwarded as-is. But this ends up not mattering in practice because you still need to (or ought to) verify the authenticity of a forwarded message (either by refetching the entire activity from origin, by stuffing the original http-sig in a Forwarded header, or by an embedded ldsig or vcdi proof). In any case, having a signature means that you can forward something arbitrarily – you give up the ability to do access control when you provide cryptographic proof of data integrity; it becomes redistributable.
It depends on what you mean by “access control”, but the forwarded object (or Announce-wrapped object) would typically still have an actor, with or without a signature. If that actor is from a blocked domain, couldn’t it be blocked regardless of an LD Signature?
Any object is redistributable, with or without an HTTP Signature or LD Signature, in the sense that it can be posted to actors on another server. However, most receiving servers will not accept messages without some kind of authentication. A typical receiving server may only accept redistributed messages with a signature, but they are not required to accept them.
Wrong way around, I think – I interpreted the question not as “how to stop seeing announces of blocked actors”, but rather “how to stop blocked actors from seeing something announced by a relay”
I understand. Thanks for the clarification. The question was vague enough it could be interpreted either way (or maybe @jdp23 was interested in “all of the above”? 
). I agree that’s an interesting issue. Isn’t that an issue regardless of Announce wrapping or signature validation on the receiving side?
I suppose this is one reason a server might decide to not configure any relays. There’s also the interesting AP-specified behavior related to public messages: that they may be forwarded to all known servers. Some servers implement that behavior. It seems like that would have the same blocking issues as relay forwarding.
There are various ways that a blocked actor can monitor public message traffic. For relay FEP purposes, should we just note that a relay is one possible vector for doing this?
More or less yeah. if you assert that you wrote something and you provide cryptographic proof, or if the thing you wrote is public information… then blocking is simply not possible to enforce. If you want to enforce blocking, then that means access control, and access control means that you can’t make it public and you can’t make it redistributable.
That sounds like it’s just about all we can do, yeah… probably in something like a Security Considerations section. Of course there are ways to relay a public message that refers to some private contents, where the private contents are subject to access control. (Referring to things by id only is one way of doing this.)
Thanks all for the discussion, as always I learned a lot …
That second version is indeed how I was primarily thinking of it but I also intentionally phrased it broadly expecting that there might well be some aspects I hadn’t taken into account.
I agree, with the caveat there are also weaker-but-still-useful versions of blocking that limit interactions (and visibility of interactions) in “well-behaved” software that isn’t intentionally trying to evade blocks. Bad actors can and will get around it, but the additional friction is helpful – Another Six Weeks: Muting vs. Blocking and the Wolf Whistles of the Internet by Leigh Honeywell | Model View Culture has some great perspectives on this. And a lot of people aren’t bad actors so won’t use the circumvention software.
But, this is a more general point … at least from the discussion so far, it doesn’t seem like relays are further weakening the fediverse’s current levels of weak-but-stil-useful blocking.
Makes sense to me. And, great discussion, I’ll think about it more and come back to it if any more ideas pop to mind.
When relay uses an Announce activity, relayed posts are processed as boosts / reposts. This is a very useful feature. For example, when relay actor is enabled on Pleroma instance, you can follow the entire instance from your personal account. FediBuzz also uses Announce, so you can follow hashtags from your personal account.
the only thing Announce is getting you in this case is that you don’t have to address your activity to a relay’s followers, only to the relay itself. inbox forwarding would ideally require addressing both the relay and its followers. there’s also the possibility of a relay Announcing your post or activity without you ever addressing it. this is what pleroma is doing in effect – your Public activities are being “delivered” to the local sharedInbox (through the internal mechanism of “querying the db”) and then that sharedInbox is being monitored by the internal.relay actor who will Announce anything it sees in there.
it would be nice to have something more controllable by users who can choose to opt into sending their things to a relay or not. in a pure C2S situation you need to be explicit. also there is the possibility of relaying private posts, for which inbox forwarding is a better fit.
Isn’t there also a difference that Announces without embedded objects must be fetched, and and Creates should be verified?
If the object isn’t embedded then it needs to be fetched, yes. But this fetching doesn’t need to happen by the server. It more often happens at the client.
Say you send an activity to an actor whose specified behavior is to Announce anything it receives. The first step here is that the relay actor’s server needs to verify that the original delivery was okay. For example, the relay actor’s server might have an allowlist of which domains are allowed to deliver activities in the first place, enforced by HTTP signatures. Separately, the relay actor might want to verify (at the client/app level) that the sending actor is allowed to relay things via that relay.
So one way that this might look like is if the sending actor sends this:
POST /the-relay-actor-inbox HTTP/1.1
Host: domain.example
Content-Type: application/ld+json; profile="https://www.w3.org/ns/activitystreams"
Signature: <a signature from the delivering server/domain>
{
"@context": "https://www.w3.org/ns/activitystreams",
"id": "https://domain.example/an-activity" // protected by access control
}
At this point the relay actor’s server can verify that the HTTP signature came from an allowed entity. It passes the check[1], so it gets added to the relay actor’s inbox.
The relay actor’s client/app then sees this activity in its inbox:
{
"@context": "https://www.w3.org/ns/activitystreams",
"id": "https://domain.example/an-activity" // protected by access control
}
And, according to its specified behavior, the relay actor can:
Each of these would be its own “relay protocol”. So the FEP may end up defining multiple types instead of just one, although certain “protocols” may be more recommended than others.
Simple and straightforward. Just take the object as-is and wrap it in an Announce.
Pros:
Cons:
{
"@context": "https://www.w3.org/ns/activitystreams",
"id": "https://domain.example/announce-with-embed",
"actor": "https://domain.example/the-relay-actor",
"type": "Announce",
"object": {
"@context": "https://www.w3.org/ns/activitystreams",
"id": "https://domain.example/an-activity" // protected by access control
}
}
Also simple and straightforward. Just take the id of the object and refer to it in an Announce.
Pros:
Cons:
{
"@context": "https://www.w3.org/ns/activitystreams",
"id": "https://domain.example/announce-with-reference",
"actor": "https://domain.example/the-relay-actor",
"type": "Announce",
"object": "https://domain.example/an-activity" // protected by access control
}
Less simple. Potentially requires fetching references until a “post” object is found, but comes with the guarantee that whatever is being Announced is a “post”. This depends on having a formal definition of what a “post” is, which would be the subject of a separate FEP, but for the purposes of this example let’s say that a “post” is any object that has content.
Pros:
object being Announced is guaranteed to be a “post” (meaning that it is immediately useful and can be rendered as-is for the most part)Cons:
The relay actor starts with this activity in its inbox, which currently does not look like a “post”:
{
"@context": "https://www.w3.org/ns/activitystreams",
"id": "https://domain.example/an-activity" // protected by access control
}
So the relay actor can follow up by fetching the activity to get more information about it (subject to access control):
GET /an-activity HTTP/1.1
Host: domain.example
Authorization: ... # whatever is supported
HTTP/1.1 200 OK
Content-Type: application/ld+json; profile="https://www.w3.org/ns/activitystreams"
{
"@context": "https://www.w3.org/ns/activitystreams",
"id": "https://domain.example/an-activity",
"type": "Announce", // probably want to limit your unwrapping to Create and Announce?
"object": {
"id": "https://domain.example/thanks-for-sharing",
"type": "Like",
"object": {
"id": "https://domain.example/the-weather-is-bad",
"type": "Announce",
"object": {
"id": "https://domain.example/miami-weather-today",
"type": "Page",
"name": "Weather in Miami for 2024-10-09",
"url": "https://weather.example/us/fl/mia/2024/10/09"
},
"content": "Hey, watch out for the bad weather..."
},
"content": "thanks for sharing!"
}
}
This outermost activity can be unwrapped if it is a Create or Announce without content. The next level down is a Like with content, which in our example “post protocol” is considered to be a “post” because it has content. So our relay actor refers to this Like in an Announce:
{
"@context": "https://www.w3.org/ns/activitystreams",
"id": "https://domain.example/announce-post-with-reference",
"actor": "https://domain.example/the-relay-actor",
"type": "Announce",
"object": "https://domain.example/thanks-for-sharing" // maybe protected by access control
}
Exactly the same as the above section, but embedding the Like “post” instead:
{
"@context": "https://www.w3.org/ns/activitystreams",
"id": "https://domain.example/announce-post-with-embed",
"actor": "https://domain.example/the-relay-actor",
"type": "Announce",
"object": {
"id": "https://domain.example/thanks-for-sharing", // if this is subject to access control, then we just leaked some info (subject to verification of authenticity)
"type": "Like",
"object": {
"id": "https://domain.example/the-weather-is-bad",
"type": "Announce",
"object": {
"id": "https://domain.example/miami-weather-today",
"type": "Page",
"name": "Weather in Miami for 2024-10-09",
"url": "https://weather.example/us/fl/mia/2024/10/09"
},
"content": "Hey, watch out for the bad weather..."
},
"content": "thanks for sharing!"
}
}
If the inner “post” is if subject to access control, then we just leaked some info (subject to verification of authenticity).
Technically any actor can forward from inbox; for a relay actor to do “inbox forwarding”, the recipients need to be defined. This can be defined in the “relay protocol” as “this actor will POST the same payload to its followers, regardless of addressees”, or it can be defined as “the payload MUST address the relay actor’s followers collection”.
In the latter case where the activity we received does not have addressing information embedded, we can fetch it to verify:
GET /an-activity HTTP/1.1
Host: domain.example
Authorization: ... # whatever is supported
HTTP/1.1 200 OK
Content-Type: application/ld+json; profile="https://www.w3.org/ns/activitystreams"
{
"@context": "https://www.w3.org/ns/activitystreams",
"id": "https://domain.example/an-activity",
"to": [
"https://domain.example/the-relay-actor",
"https://domain.example/the-relay-actor-followers"
]
}
Now the relay actor knows that the original sender intended for this activity to be forwarded to the relay actor’s followers. It can forward the activity as-is to all of its followers.
POST /some-followers-inbox HTTP/1.1
Host: follower.example
Content-Type: application/ld+json; profile="https://www.w3.org/ns/activitystreams"
Signature: <a signature from the relay actor's server/domain>
Forwarded: <a signature from the delivering server/domain>
{
"@context": "https://www.w3.org/ns/activitystreams",
"id": "https://domain.example/an-activity" // protected by access control
}
If the original sender sent a “minimal activity” to the relay actor, then this is the same case as forwarding as-is.
This option makes the original HTTP signature not usable for forwarding purposes. You need to rely on the fetch-and-verify behavior of downstream actors for this kind of “relay protocol” to be useful.
If a relay actor has the ability to Follow and to have followers, then you may encounter a situation where two relay actors follow each other, which can cause infinite reflection if you’re not careful. Therefore, relay actors MUST keep track of activities that they have already relayed, and MUST NOT relay an activity more than once. (Failed deliveries may be retried.)
Note that it’s possible that a “minimal activity” (technically a form of “thin ping”) can lead to the entire activity being discarded by servers that don’t support fetch-to-verify. But that’s mostly what you’d expect in a situation where you care about access control. In general, if you care about access control, you don’t want much more than the id to be passed around.
Certainly a lot of possibilities for combinatorial explosion here…
I think the relay actor should generally require less information to operate rather than more information, but having the relay actor have access to certain information can make it more useful.
In general:
Announce allows for potentially tracking which relays shared your activity in the shares collection, if you or your server are one of the recipients of the relayed Announce activity.I also think that it could be very helpful to think about this in terms of what any actor needs to do in order to get their activity or object relayed. So for example, what do you need to do in your client to make sure that you get the desired behavior? How do you formulate the activity? Who do you need to address? The thing about addressing the relay actor’s followers, and whether that should be necessary or not, is basically the kind of thing I’m trying to get at here. If you want to only send to the actor, then you may or may not be implicitly agreeing to certain behavior that you might not be aware of.
The exact logic of the check is an implementation detail. At the broadest level, the receiving server might assume that any key on a given domain/web-origin is controlled by the application running on that domain/web-origin. This is generally the default assumption because there’s no good way to signal that a subpath on a host is being controlled by a different application than whatever is running at the root of the host. This assumption underpins things like “host meta” and “same origin policy”. Perhaps the use of DIDs can allow finer-grained policies, depending on the security model of the DID method. ↩︎
As I do more experimentation with the Mastodon relay protocol, JSON-LD signing and verification of the messages seems to be the most complicated requirement. It requires RDF canonicalization of two documents (the message and the signature document) and then signing/verifying their combined hashes.
There are also related behaviors like “patching messages for forwarding” and checking that a message is “safe for forwarding” (both to support JSON-LD processing).
I’ve been exploring the Mastodon relay behavior further. The LD signature support is difficult to implement. I’ve been trying to reverse-engineer the Mastodon implementation and rewrite the signing and verification login in Python, but I haven’t been successful so far. I am able to create and hash canonical JSON-LD/RDF documents for the signature data and the primary document (matching what Mastodon produces), but the message signature fails verification when Mastodon receives the message. I have tests that successfully sign and verify the documents locally, so there must be something different in Mastodon that I haven’t seen and/or understood yet.
The JSON LD context used by Mastodon for expansion of the signature document seems to be unresolvable (https://w3id.org/identity/v1). I don’t know how it works in the Mastodon code. For my experiments, I load the context from a local file with a custom context loader.
Apparently, the benefit of the LD Signatures is that the relayed document doesn’t need to be fetched from the authoritative server. When Announce is used, the Announced activity must be fetched, which might be significant if thousands of servers are doing it at about the same time when the relayed message is received (assuming a large relay, of course).
On the other hand, implementing the support for Announce wrapping is relatively easy. The pub-relay server does the wrapping automatically for certain activity types if there’s no LD Signature.
I did verify that relayed Announce objects are displayed (non-boosted) in the Federated timeline in Mastodon if one is careful with the recipient addressing (the public URI must be in the to and not in cc property of the object (or in both?), for example).
Note that I’m using “relayed” in the common sense of the word, not the pub-relay definition of it. In that context, I’m interpreting their terminology as “relay/forward” (send the message as-is, which they call a “rebroadcast”) and “relay/announce” (wrap the message in an Announce before publishing it, which they call “relaying”). I also think it’s confusing if we use the term “inbox forwarding” in a relay context. It doesn’t seem like it’s the same as AP “inbox forwarding” (expanding private collections recursively, prior to delivery).
off-chance, are you accounting for the fact that Mastodon uses RsaSignature2017 which is hardcoded and not defined in any context? i think what eventually made it into the context is RsaSignature2018 instead.
this could be an oversight in mastodon’s inbox handling logic. for example this is how it handles statuses with the StatusParser:
but for delivery, code path goes through the activity handler first. in the case of both Create and Announce, there is separate audience processing done on the activity:
the audience of the activity is processed first, which in the case of Announce, is translated to the concept of “private boosts”. i don’t know how this interacts with relays because the first thing that happens in the Announce processor is to fetch the original status from either cache or origin:
i’m onboard with disambiguating like this, and i think that “relay/forward” and “relay/announce” are good enough terms until/unless we find other ones, but i don’t think it’s vastly different than AP inbox forwarding. to be precise, the mechanism (POST unmodified payload to inbox) is the same. it’s the audience processing that’s different. the “relay/forward” action seems to assume an audience of that relay actor’s followers, in all cases. so even if the relay receives an activity that wasn’t addressed explicitly to the relay’s followers, it will still “relay/forward” to its followers. this is something i’m not comfortable with, because it could violate expectations while not having a clean resolution.
i think you could call this “HTTP message relaying to all AP follower inboxes”? or more precisely, the message is just the payload and not the HTTP verb or any headers (except in the case of forwarding an HTTP signature via the Forwarded header or similar)
I found the problem. I accidentally used a “URL safe” base64 encoder for the signature. Using a non-safe one worked.
The RsaSignature2017 signature type is removed by Mastodon before hashing the signature document but there is a guard for it. Maybe it’s removed for the reason you mentioned.
The verification (and signing) uses the https://w3id.org/identity/v1 context for JSON-LD expansion of the signature document prior to RDF canonicalization.
I’m not sure I understood the point you were making, but based on experimentation, Mastodon seems to put relayed Announce activities into the federated timeline (and only that timeline) and it appears to only do it when the visibility is :public. If the Public URI is only in the cc field, the activity/object would have ':unlisted` visibility and not displayed.
If I submit an FEP, I’ll probably create one specific to the Mastodon relay protocol although I’ve seen a few other relay protocols/designs. Activity Relay supports the LitePub relay protocol and pub-relay has Pleroma-specific behaviors (mutual Follow?). However, the Mastodon relay protocol is the only one that works with both relays. We can write other FEPs to describe the alternative protocols.
I’ve submitted a (rough) draft FEP describing the Mastodon and LitePub/Pleroma relay protocols.
/cc @bumblefudge
What’s bad about it? It gives read access to their entire server, gives access to followers-only and mentioned-only posts, not just public posts.
I don’t see what I can do about Mastodon’s security issue in #FediBuzz. I was hoping the Mastodon people were using the time to make permissions more fine-grained instead of sabotaging useful services that simply help supporting decentralization.
I have added relay client functionality a year ago, subscribing to many of the other public relays. However, the resulting throughput is at 0.1 posts/s which is really poor compared to the 10 posts/s we get from the public federated timelines. Implementing it was wasted time but it totally validated the client API approach. I wonder why that is still recommended by operators of big servers as they don’t seem to bother joining relays at all, otherwise there would be much more traffic.
Please stop telling me “you are doing it wrong.” Posing as a person who wants to control a social network has strong Elon vibes. It will make me look into Bluesky and Nostr eventually where firehose feeds are public for everyone.
@Astro we are, of course, always working on ways to improve Mastodon’s security & safety posture, this is a continuous process. As mentioned in the original GitHub issue after the streaming API had security increased, the streaming API was never intended as a tool for mass data gathering nor mass data scraping without consent.
The streaming server is not intended for this usage, plain and simple: it’s intended for end-users to get quicker updates in the clients that they use.
Right now, what you’re doing exposes users to abuse that is invisible to them, but visible to their followers, because you’re accessing our content without consent and then broadcasting it to servers we have defederated from.
This is an issue that existing Relays do not create, because they gain consent to access data.
Ah, so this is about blocking? Fair enough, I myself would not want to live without being able to block stupid people on the internet. Yet, this reminds me of what people said when I grew up: “The Internet regards censorship as damage and will route around it.”
I am failing to see how relays would make a difference.