The Fediverse Challenge

[aschrijver]

In follow-up to the issues I outlined in the Mastodon versus Fediverse discussion, here I want to address an even broader, more urgent topic that I have been trying to put to attention for some time. What I find the most intriguing and challenging - and the general objective that SocialHub was created for - is not so much the individual choices of a FOSS project, but rather:

"How to keep things moving and ensure Fediverse evolves in a healthy and steady way?"

Whatever you think of Signal messenger’s choice to be a centralized platform, in the post "Reflections: The ecosystem is moving", Moxie has summed up this challenge quite well, imho:

Stuck in Time

In some circles, this [decision not to federate] has not been a popular opinion. When someone recently asked me about federating an unrelated communication platform into the Signal network, I told them that I thought we’d be unlikely to ever federate with clients and servers we don’t control. Their retort was “that’s dumb, how far would the internet have gotten without interoperable protocols defined by 3rd parties?”

I thought about it. We got to the first production version of IP, and have been trying for the past 20 years to switch to a second production version of IP with limited success. We got to HTTP version 1.1 in 1997, and have been stuck there until now. Likewise, SMTP, IRC, DNS, XMPP, are all similarly frozen in time circa the late 1990s. To answer his question, that’s how far the internet got. It got to the late 90s.

That has taken us pretty far, but it’s undeniable that once you federate your protocol, it becomes very difficult to make changes. And right now, at the application level, things that stand still don’t fare very well in a world where the ecosystem is moving .

Things that stand still don’t fare well in a world where the ecosystem is moving

In this comparison the ecosystem that Moxie is referring to in the above, is in our case the tech landscape outside of the Fediverse, where Big Tech surveillance capitalism is going rampant, steaming full force ahead at tremendous velocity.

Take the example of XMPP. A lot has happened in this technology realm. Tons of FEP’s were created, many projects launched. It still exists today and new developments still forthcoming. Some people truly love it… if they happen to know that it exists. But it has not come anywhere near to what those involved had envisioned. It hasn’t shifted the tech landscape and moved things in the positive direction to the extent that people had hoped. And though there is a recent uptick in popularity again (luckily), most devs don’t think of XMPP as an obvious choice for their new project.

XMPP may still get there eventually, who knows. It’s story looks very similar to where ActivityPub ecosystem is headed now. There’s enormous potential in the technology. But will we be able to tap into it, before something else becomes more relevant and the tech community takes a different interest? Will we and our federated AP projects end up among the many tombstones that scatter the decentralized web landscape?

Mastodon: If you don’t move, we move

As early adopter Mastodon has gained a dominant position. They benefit from it now, in that it eases their own development. Having the ‘user’ base, they can make functional decisions based on that. There’s no need to comply to any standards, or to contribute their insights back to a standards-body or community like SocialHub. It would be a win-win if they’d be more involved here, but on the short term it is not really in their interest when looking purely from the perspective of offering a popular federated microblogging platform. They have that already. If all other federated projects withered and died, Mastodon would still be around, and Eugen the last man standing.

SocialCG/SocialHub hasn’t managed to get much done in terms of standardization after the release of the initial W3C Recommendations. We mostly have open-ended discussions and things in an indetermined state. There is no common consensus on how to move ahead. All progress happens in an entirely ad-hoc manner, and between individual projects that happen to meet and agree on certain decisions. We go grindingly slow, at glacial pace.

Mastodon’s choices to move ahead in their own way, at their own speed, is perfectly understandable in this light. And each and every other federated project is doing exactly the same. The win-win of participating in this community is either not seen, or seen as too complex and a waste of time. But ultimately, imho, not doing so and build our Spiral Island together, means that individual devs will cut their own fingers eventually with that stance.

SocialHub is on a path of irrelevancy unless we act

We must find a way to empower this community or in some years, sooner or later, the Fediverse as we know it now will be replaced with the next-shiny-thing, and the cycle starts all over again. There is a “Tragedy of the Commons” that applies to Fediverse just as well, that we must overcome.

Things are fragmenting already, falling apart. Research like Spritely and DREAM is happening elsewhere, out of sight, not AP-related in particular nor put into context, and there’s zero Scaling Up Cooperation going on. We are all just as siloed as walled gardens that way, even though developing open standards that we hope will be someday broadly adopted. A grassroots movement of individualists.

To me the discussion is not whether Fediverse will be for the masses as alternative to toxic social media molochs, or just for a niche group of people that is able to uphold its unique culture. To me this discussion is about whether Fediverse has a good chance to survive in the long run. It revolves about long-term feasibility of the things we are all passionately working on.

---

Note: In this post I have deliberately worded things more strongly and alarming than maybe they really are, but that is in hopes we don’t forego the discussion while happily absorbed in our own individual projects. CC’ing @staff because you ought to be aware, imho.

(image credit: FunkMonk on Wikimedia, CC BY-SA 3.0 Unported)

[aschrijver]

A response on the Write.as forum relating to the challenge: SocialHub Community and Fediverse Futures.

---

The issue is that all federated projects do their own thing with regards to the Fediverse. They use the open ActivityPub and ActivityStreams for that, that were developed to allow interoperability such as the Fediverse (somewhat) provides.

But these standards are still immature. They contain holes that need to be filled. There’s a win-win for any project to help fill them in. Next versions of the standards, as well as common extensions need to be released. Unfortunately there’s no one really willing to help with this effort, and so the evolution of the Fediverse languishes.

This means three things:

1. Extending federation-based features is inproductive. Have to (re)invent them or learn & adapt from other codebases.
2. No single project will benefit from the full potential of the technology. It remains untapped.
3. Technology base remains weak and fragile, and there’s high risk it will be passed and replaced with the next shiny thing.

An example of the way that individual projects - here write.as in a proposal to @thebaer - tackle things is in:

Federate reader page as AP community/group - Feature requests - Discuss Write.as

Would be cool if people could not only subscribe to individual users via ActivityPub, but also to a group of blogs. The reference implementation for that is probably Lemmy, but Soapbox/Pleroma is also working on groups support. The reference for Lemmy’s implementation can be found here.

Lemmy is NOT the reference implementation of Groups! It is just one possible implementation. And it may differ from the next impl. and the next and the next. Until we have 100 flavours of Group implementation and zero interoperability overall.

That is because the Groups implementation discussions were never finished. Nothing was standardized, no agreement reached. There was insufficient interest to further open standards.

If all projects thinking about Groups support had interacted with SocialHub to share their insights, and had they written or improved Fediverse Enhancement Proposals, then the situation would much improve and there would be reference implementations, instead of ad-hoc development.

[melvincarvalho]

A few thoughts:

Definitely not a fan of signal after the MobileCoin debacle. Does fediverse have a (distributed) messenger? If not, possibly an opportunity there to create something modern, with cool new features

When we made ActivityPub in the Social Web Working group it was by no means the perfect standard, a big compromise. Trying to get something over the line. The group had 3 competing groups, indieweb, activtypub and solid. The closest we came to compromise was around JSON-LD, but that even was not consistent, different data models, different mime types, different specs, and indieweb rejected LD. So it was a bit of a mess. We didnt know if stuff would be implemented, and then mastodon came along and ‘saved the day’, in the sense that it took things to the next level.

There’s always a will to try and level up each project, but I’d suggest just enjoying each phase. There are aspects of the previous phase that you’ll miss if fediverse goes to the next level. For example venture capitalists and money might come in, try and co-opt the movement and create a whole new vibe, not always that positive. I’d say just enjoy each phase on its own merits, and keep innovating.

Regarding static vs dynamic specs. HTTP is really a very good spec, and it’s stood the test of time. As has something like TCP/IP. This is because you can do so much with HTTP and the real innovation is designed to be a layer up from HTTP. Consider facebook is built on http, and cityville was a game built on facebook that went from 0 to 100 million users in a month. This is not a one off. No other protocol has this flexibility. It’s just that some developers like change, and some like stability. To be stable it must allow you to work, however.

One nice thing about solid is that it has clean data models. Solid is however very complex, and requires you to know full RDF inside out, which can take years to learn just a fraction of that. And for most things it’s not needed. One thing solid does well IMO is the clean separation of profile pages. It has one object for the profile page (/), one object for the Person (#me) and one object for the public key (#key). It mastodon you have #main-key then profile page and person are coupled together, which is going to be a pain point as things grow. It could be fixed quite easily by add #me on the end of the “id” field in the JSON, but not sure what that would break

Another limitation of JSON-LD is that it doesnt allow you to use regular JSON so easily, because every key must map to a URI. This is too much a heavy-lift for new developers coming to things, so that’s another thing that could be fixed either in the spec, or by adding base/vocab fields to the context. To this day the W3C doesnt fully realize that developers want to be allowed to use plain old JSON with JavaScript. Once burnt by XML many take the view that ‘the serialization doesnt matter’. It does to developers IMO

So in short Id say fediverse is on a good way. Dont try and growth hack too much, and enjoy this phase. At the protocol level try and unify profiles and identity as that’s the foundation for every app. Make new apps, make a chat app (not signal please!), bootstrap existing communities, add new features such like twitter is making such as tipjars, super-follows, integration of lightning-network. Let’s make a chat app to compete with telegram, xmpp etc. more modern, and sending AP messages over the wire, in a similar format to microblog posts. Let’s make new functionality and new posts. Let’s do server to server, client to server and client to client. It can all be done with the same profile pages, which requires either some key management or delegate to an agent

For my part I’m working on ‘autonomous agents’ which will run, in the limiting case, independent of a human operator. They will have their own life cycles, be cheap to run, and be such that anyone can make one. You then add skills to the agent and it can start to do useful things. Also, anyone can make or modify a skill. In a sense it’s making servers into small distributed components without a single point of failure. I’m hopefully going to make this interoperable with the fediverse, and am working on agent profiles right now

[aschrijver]

Thank your for your elaborate response, @melvincarvalho, much appreciated. In my example Signal is irrelevant, but more the fact that once a complex decentralized ecosystem emerges it becomes nigh impossible to innovate the underlying specs and open standards, because there are many different flavours and versions out and about. Nigh impossible… unless we take this into account in early stages.

I agree on most all of the points you make, and also with the ‘just go with the fun of the current stage’. I am certainly no advocate of growth hacking AS/AP until there’s mass adoption. Just as wary of those vested interests it would attract and threat of stuff to be co-opted. (From this perspective I also have my reservations to get fully behind the Policy SIG effort to make AS/AP the chosen standard by the EU Digital Services Act. If that should happen, then the fedi community should be strong enough to handle the impact, which is far from the case now imho).

Re:Solid. I like a number of things in their initiative, but I also feel that there maybe an underlying drive to reintroduce the Semantic Web again. The complexity inherent to that is unavoidable then, and I’d worry about that as an early adopter. At minimum there’s a huge challenge to come to widespread adoption (which is one of Solid’s objectives).

For me most important is steady progress. Evolution, not revolution. And then there can be a lot of fun, since so much of the potential is as yet untapped. I am afraid that the laissez-faire attitude of “things just go where they go” of grassroots movement will eventually become a spoiler of the fun.

What I think needs to happen is not all that complex (in theory). There’s no real need for heavyweight and formal W3C processes and ceremony and creating versions X.Y.Z of the specs. There are a couple of holes still, true, and also there are existing mechanisms to extend upon the specs. Where Solid strives to support the entirety of RDF so you can define any semantics in machine-readable way, I feel that for ActivityPub we can standardize on libraries of well-defined extensions. These are just building-blocks that can choose from when your app targets a particular domain. Each extension offers concepts that can be consumed as JSON, but are also still JSON-LD. They are closed-world ‘micro’ ontologies.

What is missing is a streamlined Process to create extensions and to plug these holes. That’s all. And because the process isn’t there, most devs turn away from SocialHub and do their own thing. We made a good start in creating said process (e.g. the FEP initiative), but then somehow it stalled. Question is why, and what is needed to really get it going again.

[melvincarvalho]

@aschrijver

Each extension offers concepts that can be consumed as JSON, but are also still JSON-LD. They are closed-world ‘micro’ ontologies

+1 to this

It would definitely be possible to go this path. In fact, there’s no reason at all why plain old JSON should not be considered the semantic web. The original idea (before it got more complex) was just to put machine readable data into html pages. Today you would do that with JSON, probably inside a SCRIPT tag. This is the technique pioneered by schema.org and it’s used to give previews of links, and for SEO. It is the de facto semantic web today, with about 1 in 3 pages having some kind of form of it

[aschrijver]

Indeed, similar to schema.org, though that is all a single huge ontology, designed for universal applicability and therefore often quite abstract in terminology use. Also it is why they are very conservative in adding new terms to the vocabulary. In the more domain-oriented pattern library there’s more freedom to define your own variants, but you make a choice then on how likely its widespread use will become. I envision it as a sort of NPM for micro ontologies, where you can figure out the usage metrics.

---

Quoting from @dansup in a related fedi thread:

Implementing new ActivityPub object types unused by other projects is fun!

I don’t have to worry about compatibility, and I get to define my own object schema #pixeldev #activitypub

My response:

Not particular to your efforts, but in general:

There is however the person that comes after you to consider. Should they contact you? Dive into the codebase to figure out what you did?

A big challenge for fedi is how to get alignment and consensus on the interoperability constructs to use. Right now its an ‘everybody for themselves’ kinda thing.

And subsequent reaction:

I agree. I plan on adding federation documentation.

My long term goal is to adopt a machine readable ActivityPub implementation schema for FediDB with crowdsourced data for each project so one can build and test against real world objects.

Delighted to hear this, @dansup. Main keyword here is “crowdsourced”, as there will be Process involved…

@aschrijver

What is missing is a streamlined Process to create extensions and to plug these holes.

How can we create said process so that an inclusive, active and open community emerges that productively creates these extensions? A community where cross-pollination occurs and we collectively drive each other onwards and upwards?

PS. Note that for @cjs Go-Fed and the GoToSocial impl. based on top of it, a discussion is ongoing how to support extensions. Ideally we should bring insights and discussion to SocialHub and related to other subjects, such as FEP process.

[cjs]

@aschrijver

PS. Note that for @cjs Go-Fed and the GoToSocial impl. based on top of it, a discussion is ongoing how to support extensions.

The “how” has been long known to go-fed: Write a JSON-LD file (example forgefed) containing OWL descriptors. Boom, it is ready to be federated itself over ActivityPub! It is of a similar level to OpenAPI/Swagger, so that it can be machine-readable and therefore code-generated from, or presented in a documentary web site in a consistent manner.

The tough part is that no one really cares if your background is in dynamic languages (a lot of fedi projects) and if RDF thinking isn’t constantly top of mind (it generally isn’t even for me).

[melvincarvalho]

Yes, it’s quite hard to get terms into schema because you have to demonstrate wide usage, then they go in as ‘pending’

I suggest a “developer first” solution which is an incremental approach that could work for the whole ecosystem

1. New terms that you’d like to try out, just use as plain old json keys.

2. As terms become more mature you can start to add them inline to the context, to make them more stable, better described

3. Eventually you can move things out from the inline context into their own vocab area. I’d suggest very lightweight vocabs with just the term name, and a description

{
  "@id": "#term",
  "http://www.w3.org/2000/01/rdf-schema#label": "term",
  "http://www.w3.org/2000/01/rdf-schema#comment": "Term Description"
}

Note: the relative URL in the @id which does not tie the term to any particular domain

As JSON-LD: JSON-LD Playground

It’s possible to get extremely complicated about vocabs (AS is a bit of a mess tbh!) but ultimately they can be quite simple things, with a namespaced term and a self description. The other things that people add are optional, and rarely used

Ultimately you want to publish it somewhere too. This could be at a place of your choosing or perhaps, it could be fun to get a shared area, something like:

https://w3id.org/

e.g. under the namespace /fedi/

IMO a really cool way to do this would be say to have /fedi/pixeldev/ (note the trailing slash) then just dump in index.html with a script tag that contains your JSON, and you can mark up the html however you want, or not at all if preferred. This is perfectly valid JSON-LD 1.1

Regarding publishing: normally the shorter the URL the better, and the longer it’s around the better

Could be a nice joint area for different projects to collaborate and share patterns.

[aschrijver]

The discussion has taken an interesting turn into Process and Extension mechanisms. With the related discussion on Pixelfed Groups we might split this into a separate discussion.

On the original broader topic of The Fediverse Challenge I just wrote this follow-up on the new Fediverse Town forum targeted to more non-technical fedizens:

The Fediverse Saga: How Shall We Conquer Our Challenges?

[aschrijver]

I am more or less burning out on advocacy for improving fediverse open standards and a healthy community that builds them, but since I posted this already in a response to a discussion between @dansup and @cjs, quoting here:

A federated protocol where everyone creates ad-hoc extensions for their own projects means we’ll soon have a spaghetti-code #Fediverse that no one fully understands and gets harder and harder to interoperate with and less attractive to build new apps for.

The tech debt is already very large, and most documentation is lacking. The “Tragedy of the Commons” of fedi is that it is not a project in and of itself, so nobody maintains it, only uses it as a dependency in their own project.

What I didn’t mention in the toot due to the char limit, is that with too much tech debt a project will grind to a halt and die. Or restart from scratch, which in our case probably means people jumping to a new technology. I notice that already with a huge uptick in Matrix protocol popularity for all kinds of social app use cases.

In the linked thread there’s also a call for:

• An improved standards track that devs can be involved in, in parallel to their projects.
• A more formal organization that underlies this community (I had Fedi Foundation in mind for that).
• Monetary incentives via funding to encourage people to do non-project-related community work (i.e. the chores).

[weex]

As you know I’ve been cataloguing fedi problems for a few months but honestly tech debt never came to mind. I see many projects that are frozen in time as their owner/developers moved on but tech debt is something else, the result of kicking the can down the road, applying band-aid solutions, duct tape and bubble gum. In other words, never refactoring anything so that it’s more efficient or more beautiful than use requires.

I’m fairly comfortable with the spaghetti and the messy evolutionary processes that happen in decentralized technologies, but I also see standards and incentives as balancing forces that keep us moving forward. I choose to spend time on protocol and testing standards which isn’t without its own challenges but I think fits better overall.

I guess part of my problem-focus is to try and prove where standards and incentives need work. This takes the Challenge from one of advocacy into a set of interesting problems to work on, that we’ll be better equipped to solve by developing and using the right tools together.

[aschrijver]

@weex

I guess part of my problem-focus is to try and prove where standards and incentives need work. This takes the Challenge from one of advocacy into a set of interesting problems to work on, that we’ll be better equipped to solve by developing and using the right tools together.

Yes, but that wouldn’t be enough. You only need to parse the history of this forum and you could fill a whole issue tracker with stuff to address. But it would only result in that you created a copy in yet another location.

There should be people willing to really be involved with elaboration and going from problem to solution. In reality you’ll find little interest to do this, other than the ocassional person giving a response. To keep something going now you have a full-time job pinging people, asking feedback, organize events, and to document in the proper place or doing it yourself, etc. etc. People like to code on their own projects (understandable) and not do the boring chores and persistent activity that’s required to keeps things going.

[weex]

@aschrijver

There should be people willing to really be involved with elaboration and going from problem to solution.

Agreed which takes me back to incentives. I wonder what drove so many to create their initial fedi-visions and how that might transfer into broader interoperability.

[how]

@aschrijver

I am more or less burning out on advocacy for improving fediverse open standards and a healthy community that builds them

Please take some time off if you must. You’re much more useful with a happy face than a crippled back and destroyed morale. The world will wait for you and keep running you already did a lot, you can let people digest it and take position, without fearing of missing out.

[aschrijver]

Thank you, Hellekin, appreciated! Things are not that bad I am just shifting attention to do more of the things I like to do myself.