I will look at the specifics of the layer model later, but want to mention a deeper problem related to the subject first …
ActivityPub needs a consistent domain model
Discussions like these in various shapes and sizes have been going on for years. They always find traction that is just so-so and part of the reason is that everyone has their own understanding of terminology and language being used. If it comes to consistent language and abstractions, the AP world represents a jungle. 
I also agree on the need to be able to distinguish between different layers. As I argued before on this forum, in the thread on the 3-stage bottom-up standardization process: We need one consistent ubiqitous language when we discuss protocol capabilities and requirements. Or else anyone leaves any discussion with their own idea of what was just discussed.
Ubiquitous Language (UL) is the terminology that all stakeholders agree to use consistently when referring to concepts of a particular domain so that there is a common understanding of the domain.
Or in a picture that looks like this:
(source: infoq: ddd context mapping)
In the UL we should answer “What layers does the protocol have?”, “What is an Extension?”, “What is a Vocabulary?”, “Are these these properly named, or can we improve for better common understanding?”, “What is the complete set of terminology we use, how do things fit together?”.
Update: I posted a toot mentioning additional challenge where resistance to change should be overcome, where app focus prevails over ecoystem health concerns.