I'm building fediverse.to!

Nope, there’s instances from all over the fediverse, including Pleroma, Misskey, etc. I don’t know which software each instance is running since I’m not polling/scraping that info, but it’s on the feature list.

1 Like

is it a crawler? what are you crawling for ?

Currently I’m fetching the metadata API for each instance. It gives me user stats, language, description, etc. Categories I fetch from the instances.social API plus pattern searches in Python.

Been doing this for over 10 years, we have an API etc if you need any easy ways to populate all servers etc. https://fediverse.observer

3 Likes

This one ?

Your API was already on my list. Thanks for building it!

That’s right. Also using the API on mastodon.social.

Hi all :wave:. I have an update on fediverse.to. I’m now tracking 84 other fediverse software which are all searchable.

I need some advice on how to categorize instances.

I’m currently using a list of limited keywords to group instances into categories, but the keywords list is very limited and based on the categories that instances.social used to return in the API. I’d like to improve this feature by adding more keywords and categories but I’m not sure how to do that. Is there a list somewhere on general categories and keywords that I can use? Is there a smarter way to go about this?

Any and all advice is appreciated. Thanks :slight_smile:

1 Like

FWIW, we just removed all of our tracker support. Some of us came here to get away from being the product. We’re just people communicating with other people and it’s frankly nobody else’s business what software we use or what we like to talk about. Anyway, not a problem. Don’t come looking for us because you won’t find us, and if you do, we’ll be sure to prevent it from happening again.

1 Like

Once I got annoyed when I found out that over half the traffic to my single-user dev site was Friendica instances trying to figure out what kind of software I was running by repeatedly probing for APIs and interfaces from a huge list of those they knew about. I ended up writing a plugin to redirect all of these requests back to Michael Vogel’s site. I suppose the moral of this story is that if a site isn’t running Nodeinfo, just leave them alone.

1 Like

For example we display the server system of a contact to show the diversity of the Fediverse. Also not all group implementations are done in the same way, so we might end up in having to implement target specific content. (By now we are excluding Peertube groups from our forum list, since these groups aren’t groups in the way other implementations are using them). Also we might in the future implement some Lemmy specific stuff (like their spoilers). For these things we need to know the remote system.

Since not all systems (and versions) support Nodeinfo, we have to query stuff like the old statistics.json and some implementation specific endpoints. Especially when those endpoints don’t exist on the system, they shouldn’t create a measurable load on a system.

However, in the future we could implement a mechanism that queries the robots.txt first and looks specifically for some “friendica” entry there.

display the server system of a contact to show the diversity of the Fediverse

ack, it is also important cause of “compatibility” layers for software not following the Conformance Section (not supporting “the entirety”). Personally my software would like to know what it can send:
ActivityPub or only Note or only Banana …
The standard has a generator field.

All:
In the last 4 meetings, Feb.-May, we talked about groups.
The parties who attend the meetings will do it together and interoperable.
The next meeting Developers Meeting 7 : A standard way to query in ActivityPub and we can also talk about “Potential Actions” and how to self-describe workflows.

Yeah, we are providing information that generator field.

First time I hear about statistics.json. Do you have a link with more info on it?

(OT: This ever growing list of incompatibilities to take into account seem to me unsustainable in long-term, and true interop being on a slow ‘death march’ if we don’t find ways to improve our practices)

2 Likes

The statistics.json had been introduced by Diaspora. They replaced it with the Nodeinfo, but some older systems still exist that use this endpoint.

BTW: I don’t think that the incompatibilities are growing. Most detection problems occur with older system.

1 Like

It wasn’t one Friendica site hitting some dead endpoints every week or two. It was every Friendica site checking the same 10-20 endpoints over and over and over again - every few seconds; all day, every day 24/7/365. To be fair, this was a year or two ago and I haven’t observed this anti-social behaviour recently.

Nowadays these sites are much more difficult to categorise. Every site determines their own software project and brand and they are much more resistant to data harvesting.

I’m curious, what’s so different between fediverse.to and fediverse.observer that both exist?

The Observer has been around for a lot longer and tracks most, if not all, fediverse instances. Along with the standard server information, it even lets you find a server close to you using a map. It also has an API that other devs can use to build their own apps.

Fediverse.to (my project) is aimed more at non-technical users, so there is more focus on visuals, full-text search, searching by categories, languages, etc… I’m currently working on improving categories and adding support for instance configs like maximum post characters, maximum polls, etc. I also want to add visual icons and descriptions for the different fediverse software but finding that info is proving a bit challenging.

In a lot of ways, the Observer is a superset of my project. I don’t use their API yet, but I’m considering using it to enrich the language and location of instances that already exist on Fediverse.to. And my project doesn’t currently link to other fediverse projects, but I’ll add outbound links to the Observer and the Fediverse Wiki in the future wherever appropriate.

Hope that helps :slight_smile:

3 Likes

@Sebastian this is what i meant in this thread

Well, the ActivityPub specification says to support at least all ActivityStreams types which is what redaktor does. This is in the Conformance Section and the endpoints field on the Actor object contains the OAuth endpoints but unfortunately was not specified further for AP. Even after Aarons nice video OAuth 2.1 and ActivityPub - ConfTube