The immediate fix for this is to discard a Note if I don’t have the Note it’s in reply to (instead of creating a new topic) (PR for that is here), however I know that some implementations try to “walk up” reply chain.
I guess I’m thinking out loud here as to how much reply chain walking makes sense in a forum context. I mean ideally we have a collection in a context property to work with, and the Discourse plugin actually checks this already, but we don’t live an ideal world. This is what I’m currently thinking for reply chain walking:
Go back N number of replies (perhaps 5) to see if there is a Note already associated with an existing post.
If we find a Note (say at the 4th iteration) we import ALL of the intervening Notes, and add ALL notes as new posts in the relevant topic.
So we’d end up with 5 new posts in the existing topic in this example.
seems reasonable enough, at least in the absence of a clear signal that posts should be grouped together (i.e. context), you end up having to handle contextless posts somewhat differently. walking up the reply chain to an arbitrary depth is one way to do that. i’m guessing at least 3, but 5 would probably work too.
It looks like in most cases the parent object was present and was a Note, but ActivityPub plugin failed to fetch or parse it.
In particular, replies from Streams are always breaking the thread.
Yeah, I didn’t mean to suggest the inReplyTo object wasn’t available.
Just that if it’s not already present locally, or it doesn’t have a context, the plugin is currently creating a new topic with that Note. Hence the need to traverse the reply chain to determine if there’s an existing topic to put it in.
I guess the point is here that reply chain traversing in a forum is more specifically for the point of topic detection and curation (making sure the right Notes end up in the right Topics). A post cannot exist outside of the context of a topic like it can on social media.
Specifically, NodeBB doesn't provide a context right now because there was movement on that area and I simply opted to wait.
I could provide a resolvable context. Right now if you query a NodeBB topic, it returns as:Page, but if Discourse can handle an OrderedCollection, I will do that ASAP.
Especially if @trwnh@mastodon.social can confirm that that's roughly the direction they're thinking of as well.
@angus@socialhub.activitypub.rocks, specific to (streams), I discovered last week with @mikedev@fediversity.site's help that his content was failing NodeBB's key ownership checks because (streams) uses the equal sign in their HTTP signature values.
I was naively doing a .split('='); and accidentally discarded part of the URL that requested it return a public key
you can set the same context. this is the equivalent of posting in the same topic.
you can set a new context. this is the equivalent of starting a new topic.
you can set no context. this is the equivalent of having no topic.
the caveat is that there are some posts that have no context/topic but are still intended to be in the same context/topic. it’s up to implementers to decide which heuristics they want to use for implicit inclusion. inReplyTo chains is one way.
the assumption is that a context/topic is owned by only one server, but it is possible to e.g. use alsoKnownAs or aliases or similar in order to assign multiple identifiers to the same collection.
i realize the “peering agreement” bits are still not written down, but you can have the context collections follow each other. that way, they should stay in sync (assuming no delivery failures).
So in that case would I be incorrect in changing the context to the local NodeBB context collection?
I was thinking specifically of this line in 7888:
You MAY set your own context, if you wish for your object to be in a separate context owned by you.
Which I read as "I can set my own context in parallel", but I realize now you might've meant for that to read "separate context in the case of topic fork/split"
i meant the latter but you could probably do the former if there were a mechanism to link together equivalent contexts as aliases of each other. for now, the easiest thing to do would be to just copy the “authoritative” one by whoever created the thread.
One thing that is not currently done is what we talked about here, inheriting the authoritative context and serving that instead. I will need to think that through a bit more.
I can see an argument made against doing so if it locks up the process, but the downside is you'd still have some cases where you don't get the full context.
Either way this may be moot if an iterable context is found, so inReplyTo traversal is ideal as a fallback mechanism.
Edit: in NodeBB's case, we call an internal recursive method called getParentChain which just makes the S2S call and adds it to a Set. The method terminates when it encounters an object with no inReplyTo or is unprocessable.
The honest answer is that a limit makes some intuitive sense to me, but I have medium to low confidence in the cogency of my thinking on both the limit and where it’s set. I’ve set it at 3 as that seems to be the more “conservative” (read “safer”) approach while I think it through further / see how this first version works in practice.
In terms of the “risks” (to the extent they exist) I think I’m thinking a version of the following:
You could be sent a random Note inReplyTo an unrelated Note that’s part of a large chain which you end up traversing for no reason.
Even if you eventually get to a Note in an existing topic, say 20 replies in, is it still right to say that those replies are part of your topic in a coherent sense? In what scenario would you be missing 20 odd replies? Perhaps there is one.
I guess one of the things I’m assuming is that other services are implementing the Inbox Forwarding spec correctly, which would mean that, in an ideal world, you should already have the replies you should have anyway and this is more of a “stop gap”.
However, I note that Mastodon violates the spec here, which means that more replies from Mastodon might be missed than is ideal
more replies from Mastodon might be missed than is ideal
You are not incorrect. In practice the following situation happens occasionally, especially in larger/busy topics:
You post a reply to a topic/thread (branch A), but a different branch (B) of the topic occurs outside of your view (since the activities are not forwarded to you)
Later on, someone you do follow replies in branch B, and you receive it.
Traversal finds 20 posts in between you missed, and they are all added at once, and you receive the notification of new posts in the topic, except now all of the "new" posts are scattered throughout the linear flow
Additionally, some of these new posts might appear in places higher up than where you last read
So this violates the assumption (at least in NodeBB) that if you have a "read up to" point in a topic, that there will not be new content above that point.
is it still right to say that those replies are part of your topic in a coherent sense?
From a purely technical point of view, yes, they are part of the same context (at least as derived via reply chain traversal), but from a UX POV, you could make that argument.
A forum with a linear flow of posts tends to diverge less often due to the nature of the presentation of posts themselves; something threaded models don't need to contend with.
You could be sent a random Note inReplyTo an unrelated Note that's part of a large chain which you end up traversing for no reason.
Another legitimate concern. My counter is that traversing the chain is rather inexpensive: XHR => (do other things while waiting) => inReplyTo? XHR... etc.
Actual note processing is done only once the chain is complete, and a positive relation is found.
... but I can see how this could lock up the process in other languages where processing literally stops when waiting for the XHR to complete.
Yeah, I agree with you on both points. It’s similarly inexpensive to make 20 requests in Discourse (it’s in a background process on a seperate thread).
On reflection I think part of my conservatism here is that I don’t like that Mastodon doesn’t forward activities properly and I don’t like starting from a position that I need to import 20 notes that Mastodon failed to forward to me It causes similar metadata issues in Discourse.
It causes similar metadata issues in Discourse.