Backfill and the 429

Software Discourse
1

I ran head-first into this when I was POSTing stuff to SocialHub (sort of a manual reverse topic backfill) a couple months back.

I just tested this and found that Discourse will still send a 429 if ~15 connections are made within the span of maybe a minute.

If we’re looking to do topic backfill this could be problematic if NodeBB tries to query Discourse > 15 times via AP. We’d likely do this in parallel and fire off all the calls at once (or as fast as the systems lets us wrt open connections).

I don’t think there’s anything specified in the AP protocol about querying multiple endpoints and having results batched, unfortunately :slight_smile:

2

Alternatively we could just drop all the objects we can’t retrieve and try again later. I think that’s likely the more fault-tolerant solution overall, but it’d be nice for AP GETs to not be rate limited…

3

Honestly I feel that rate limiting should not be removed. If this becomes a standard, it would soon become a well known attack vector.

1 Like
4

Fair enough. Raising the limits shouldn’t be that big of an issue though I presume?

I’m just thinking about performance re: backfill, and what any potential bottlenecks would be.