The AS2 specification describes it.
the actor is interested in any activity performed by or on the object
In the case of following a non-actor, the "on the object" is the key phrase. When some actor performs an activity on the object, that activity may be delivered to the object’s followers collection (subject to authorization, some activities may be considered private-ish).
The ActivityPub spec focuses on the “performed by the object” part, but doesn’t prohibit the “on the object”.
Note that attributedTo is not required to be an actor. AS2 says this explicitly.
Identifies one or more entities to which this object is attributed. The attributed entities might not be Actors. For instance, an object might be attributed to the completion of another activity.
I’d think any activity performed on the non-actor object by an authorized actor would be delivered to the followers collection.
EDIT: I think they have recently modified it but FunkWhale supports (or did support) following a music Library object, which is not an actor, so there is an implementation precedent for this type of behavior. IIUC, they changed, or are changing it, to be more interoperable with the Mastoverse.