This is what solidarity looks like
https://thenexusofprivacy.net/what-solidarity-looks-like/
(Part 2 of “Decentralization” and erasure: Blacksky, Bluesky, and the ATmosphere)
1 Like
@thenexusofprivacy @general @fediverse @fediversenews
Like this because it celebrates the truly independent groundbreaking activities of Blacksky on AT Protocol while clarifying that the network still has a long way to go to be effectively decentralized due to the overwhelming dominance of Bluesky PBC. It is possible to both cheer for these independent efforts and hold the company to account.
Really enjoying this article from @thenexusofprivacy about #Blacksky and what is cooking in the independent part of the Atmosphere https://infosec.exchange/@thenexusofprivacy/115186867637585567
@berniethewordsmith thanks, glad you liked it!
@thenexusofprivacy Have a follow-up question. Infrastructure wise, is it still more expensive to build an instance there compared to Fedi? Like, is Blacksky able to operate 100% apart from them or do they still rely on the corporate servers for some stuff?
@berniethewordsmith Good questions! Right now Blacsky is still using some important pieces of Bluesky infrastructure: the AppView, the platform-level automated moderation for CSAM / malware / spam (which Bluesky outsources to Hive), and the PLC directory (although they're running their own mirror of that so paying the costs for it). https://neuromatch.social/@jdp23/115107435947310644 talks a bit more about dependencies.
I'm not sure of the costs. A big difference from fedi is that the PDS only stores information from people hosted there (as opposed to here where instances have copies of all the info that's sent there). On the other hand Bluesky and Blacksy support uploaded video so it probably takes more per-person. The AppView (which they're working on) is something of an unknown -- besides Bluesky's, the only other ones currently active have just a handful of people, so it's not clear how costs will expand. Replacing platform-level moderation is expensive; Rudy estimated $160K/year -- although this is a place where the direct comparison to fedi doesn't make sense, since most instances don't do that. There are starting discussions with other projects about how to reduce costs on this, but it's early days yet.
@thenexusofprivacy @general @fediverse @fediversenews
With no extra context, I found nothing wrong in Jay quoted post.
It's good to see other server/infra/organizations taking their independent choice, though.
I wonder if Mississippi/UK rulers will understand this, and point their censorship requests to the correct organizations.
The other day, we saw Microsoft asking a Fediverse admin to erase an account cached from some other server ...
Yeah, I wasn't trying to get into a debate on Bluesky's moderation policies in this post, just trying to highlight that Blacksky is able to take a different.
Some context though is that Bluesky moderation changed their guidelines to prohibit "glorifying violence" -- then ran amok and started deleting a lot of posts that weren't actually glorifying violence, they were just very unsympathetic to Kirk. Is "rest in piss" glorifying violence? I wouldn't have said so! But hey, it's their network, they set the rules.
In terms of Mississippi and the UK, it's hard to know what regulators will focus on. I did another post last month about the Mississippi situation - https://privacy.thenexus.today/mississippi-bluesky-blacksky-the-atmosphere-mastodon-and-the-fediverse/
I had seen that letter from Microsoft asking to delete the account that was federated in from another server. As decentralized social networks get more traction, I think we should expect to see more of these. I don't know the specifics of the laws on this, so hard to know whether "it's just a cached copy of the data" is going to be a successful defense. In general though I think both this and the age verification situation are areas where there's a lot of value for fedi and the ATmosphere to work together to influence regulations and legal interpretations as things move forward.
More on the Blacksky costs: I asked Rudy and he says the PDS is running on a digitalocean storage optimized droplet at $262 /month and expects it could scale to a million users -- Bluesky scaled their PDS to 500K users before they split it. As of late August they had 328GB of blob storage; it's $0.02 / GB for more.
Relays are cheap these days, I don't know about Blacksky's in particular but in general it's around $30/month or less. Once they do an AppView it's hard to project how expensive that will be; there aren't a lot of independent ones yet beyond a hnadful of users.
@luca @general @fediverse @fediversenews
@berniethewordsmith
is there a source for this number? i highly doubt $30/month covers the traffic for the entire network over all of time. you could maybe achieve this by relaying a subset of all pds data, or only storing the last day/week/month/etc. maybe the architecture has been reworked to offload most of the costs to the appview instead?
doing a google search, i found A Full-Network Relay for $34 a Month | bryan newbold (🇪🇺Europa again) | WhiteWind blog which claims that same $30/month (or $34/month with tax), so i assume that’s the source of the claim… but it also claims the default backfill is 72 hours, and even talks about reducing it to 12 hours or as low as 1 hour. from the configuration snippet a bit further down, we see that the chosen window is just 2 hours:
# configure a short 2 hour backfill/replay window RELAY_REPLAY_WINDOW=2h
and then you need to bootstrap from bsky.network if you want it to be useful, because most people (or appviews) aren’t going to be using your relay specifically. or otherwise you need to discover and add your own pds sources.
if 2 hours of bsky.network data costs $30/month, then what does the past 2 years cost? the “previous” article Notes on Running a Full-Network atproto Relay (July 2024) | bryan newbold (🇪🇺Europa again) | WhiteWind blog claims that $150/month wasn’t enough to backfill the entire history:
trying to backfill the full network with default configuration was resulting in OOM errors with an instance this size
that’s with 12 vcpu, 32gb ram, 2tb nvme storage. the article also appears to claim a 500,000 repo limit:
RELAY_DEFAULT_REPO_LIMIT=500000
so it’s commendable that their “relay sync 1.1” managed to reduce costs of relays by removing the requirement to cache entire repo histories, but in return it seems that the costs have been shifted elsewhere (the appview?), and the costs seem to be incurred anyway if you want a “complete” view of the data (i.e. if you’re going to store it).
footnotes/addendum:
- you can’t enumerate all identities. you can enumerate all did:plc identities and discover pds services that way, but did:web is inherently not enumerable.
- you can choose to relay something other than “everything from bsky.network”, and arguably this is the more interesting approach to take, especially if you are trying to establish a more limited community. relaying a subset of data is interesting because you get to decide which subset is interesting.
- the relay alone is arguably not the most interesting part to host anymore, especially since history storage seems to have been split off. perhaps cost analysis needs to look at the relay+appview instead of just the relay? unfortunately i can’t find detailed information about appview costs right now.
Yeah, Bryan's article and another that I can't find now were the source for the "around $30/month or less". That turns out to be a bit of an underestimate for Blacksky; I found Rudy's post from when they launched atprotol.africa back in May:
" 2 vCPUs, 8GB RAM, 66GB storage for PLC directory cache + ~200GB per 24hrs of backfill replay window
Running on a $66 per month bare metal OVH instance that’s almost overkill tbh"
In terms of backfilling the entire history at the Appview, buture.blue said in July that it was 16 TB, so the AppView cost including storage was $200. https://whtwnd.com/futur.blue/3ls7sbvpsqc2w Of course that's only one component of AppView cost, there's another component that scales at least linearly with number of users, so like i said it's hard to project.
Agreed that subnetwork relays are interesting too and really though that things would move more in that direction in general but I think the appearance of additional full-network relays (and Bluesky doing most of their moderation at the AppView and app levels so far) has taken away the time urgency of exploring that path. Projects that are just starting up are fine using Bluesky's relay knowing they can switch; projects that are at scale and seriously investing, $100 month or whatever for the whole-network relay is no big deal.
1 Like
say no more. i’ll join fight the good fight.
i was steering my own project into fix a very annoying thing (moral bankruptcy) in China. I might as well do it openly now. shits happening in tech’s gone completely stupid.