I’d be interested in hearing more about practical examples of “implementer madness”. So far I am unconvinced that it is particularly problematic to use attributedTo on Activities. In fact, I am having trouble thinking of other potential uses for attributedTo on an Activity that aren’t already covered by the use of actor on an Activity — like, does it make sense to “author” an Activity on behalf of some other “actor” perhaps? This example usage feels like it would be best served by a different extension property, perhaps author as you suggest or perhaps something already existing like dc:creator.
To me, the most natural semantics are “(This activity) is attributed to (some other activity).” It gets a little hairy because actor is considered a sub-property of attributedTo… which means you can infer that the Activity is attributedTo the actor, since actor implies attributedTo. To be explicit, this implies that attributedTo has multiple values: one for the actor who performed the current activity, one for the activity that resulted in the current activity. But it would be redundant to declare both actor and attributedTo in every single case. So implementation-wise, you would be able to get away with it, as long as you could identify that attributedTo points to an Activity.
I am not sure we need a resultOf property with largely the same semantics as “attribution”, but I do instead see a separate complication for implementers — what if someone uses attributedTo on Objects to point to the Create activity that was associated with the object? That seems like a valid and perhaps even more fitting interpretation of “attribution”. This runs tangent to the other debate about whether we should be doing all our “post” logic at the Object level or at the Activity level. So again, it looks like authorship is the semantic use case more in need of an explicit property. Madness? This is ActivityPub! 
This would be convenient to have but IDK about making it required. We are definitely missing a “follow your nose” way of going from a replies or followers or following or likes or shares back to the object or actor they belong to. Less so for context, because you’d have to keep track of context URIs anyway.
Maybe we can do this with attributedTo? Assuming that we say a Collection is “attributed” to an object, that is to say, the object resulted in the creation of the special collection, in the same way that the Create activity resulted in the creation of the object. It’s attributedTo all the way down. (I’m only half-joking!)
If there’s anything to blame here, I’d point squarely toward the circular definition of the attributedTo property. At least with this thought experiment I’m being consistent with my interpretation of “attribution” == “resulting in creation of”. It’s simple causality. If you look up the definition of the word, that’s the sense you get:
attributed (verb)
regard something as being caused by (someone or something).