Federating real-time “rooms” (presence + live audio), not published objects…what am I missing?

ActivityPub Server to Server
1

FEP-be68 (audio objects) got me thinking… and I’d value this community’s eyes on an adjacent problem before I go down a possibly-wrong path.

FEP-be68 / Funkwhale federate audio as objects, as in recordings, albums, libraries: published artifacts, distributed async.

I’ve been working on something maybe adjacent: federating real-time “places” — rooms where people are present together, live, with audio and concurrent events.
Not “publish a recording” but “two instances, jointly rendering one room, while people are present in it.” (The party line is: “Presence, not posts.”)

A few things make it seem like a genuinely different problem rather than variant:

  • Presence is the payload. The state I’m trying to pass is “who is here, live” — ephemeral, not a record with an id to dereference. Much of AP’s machinery assumes a durable object; a live room mostly isn’t.
  • Federation runs during the experience, not after it. A room handed off or mirrored mid-session can’t surface seams to the people inside it.

My instinct is that a chunk of this can’t be retrofitted onto AP’s object/actor model and wants its own minimal protocol — but I’d much rather be told I’m wrong, or pointed at people who’ve already thought about it.

Here’s the concrete Qs:

  • Any prior art for federated real-time presence (not messaging) I should be reading? Anything in the XMPP/Matrix lineage that carries over?
  • Where would you draw the line between an AP extension and a separate protocol here?
  • Any naive mistakes I’m about to make?

Grateful for any pointers / happy to share more about what I’m building if useful.

2

Welcome to SocialHub!

XMPP is the eXtensible Messaging and Presence Protocol. ActivityPub is a protocol for publishing activities (and sending them as notifications). I suggest reading the XMPP RFC’s section on presence.

It sounds like what you’re trying to do is establish a session, not publish a resource. What you probably want is something like WebSockets, but you’ll need to make sure everyone in the session is using the same identity system.

If any mistakes are about to be made, it would be trying to use ActivityPub as an “everything protocol” for things it wasn’t designed to do. There’s plenty of working deployments of XMPP for this use case, but the vast majority of them just happen to have federation turned off.

If your goal is to have a sort of “live audio stream”, the audio can also be carried by other protocols such as RTSP or Icecast – if you use WebSockets, you will need to handle chunking/encoding/buffering yourself inside your application layer.

1 Like
3

Perhaps you just need to use activities other than Create / Update / Delete?

Check out this project: GitHub - MaddyUnderStars/shoot: ActivityPub federated instant messaging server. https://shoot.pub · GitHub

They work on real-time text, audio and video chats over ActivityPub.

2 Likes
4

Thank you both, this materially reshaped my thinking.

@silverpill , I took the Shoot pointer and checked out the code, which answered a lot of puzzles. In particular, the whole thing turns on the Join/Accept handshake (Join to room; Accept carries media token + signal server’s addy; live session drops off AP from there). You unlocked it with “use activities other than CUD” cause I was assuming presence needs to live in the object model…but the verbs can carry the session.

One thing I found tracing the media handler (if someone esp closer to Shoot knows differently, def correct me): presence there is owner-only. voice_state gets written on the instance that owns the room, “who’s here” comes from that local table, and there’s no AP-level occupancy…so nothing federates the room’s presence outward. Is that simply the desired shape, or has someone tried modeling federated occupancy as a first-class thing on the AP side?

@trwnh: “establish a session, not publish a resource” is helpful. I’m now treating AP as doorman (identity + permission to enter) so a separate real-time layer can own everything inside a room.

Now I’m trying to figure out this part: you said most XMPP-for-presence deployments run with federation off — why is that? Abuse surface, operational cost, presence leaking across s2s are my guesses but does it bite a “room” the same way it bites roster presence?

Going back to XEP-0045, a MUC room’s occupant list is owned by a single service: the room lives at room@service, and that service reflects presence out to all occupants, remote ones over s2s. But Shoot keeps the same authority shape and doesn’t broadcast out.

So I’m leaning towards one home instance per room as sole authority (same thought about hosting, meaning one party responsible for the room), Shoot’s Join/Accept for admission, and the home fanning occupancy back down to the instance whose users are inside. Roughly MUC’s shape, rebuilt on an AP doorman. Therefore federation-off (or not) matters…if it’s problematic in practice I wanna know how come.

Either way, thank you both!

Eric

2 Likes
5

My kneejerk reaction would be to extend AudioObjects to Stream types and work with URI’s to actual live streams of (buffered, but minimal) audio.

Your extension could even drop a few properties if you leave that to the middleware:
You could let the client and server do content negotiation on the kind of stream (encapsulation and codecs) that are preferred.

I am working on a related project, I would love to collaborate!

Creating Presence of a Person could be something that can be Deleted for example, no need to pull in another protocol!

6

Hi, maintainer of Shoot here. The federated calls aren’t done, which is why there’s no outward federation voice state yet. I hadn’t thought about it, but my naive solution currently is just to have a collection on the channel available with the voice state, and maybe the Accept activities can be Announced to other instances.

I agree with trwnh, Shoot uses AP to just establish the webrtc session rather than actually track it or put call data over it. I have lots of issues with how Shoot currently does it though…

2 Likes
7

Thanks. I added my thoughts on the issue re: vocabulary choice.