Why not? This is a fundamental freedom of federating applications. You’re not responsible for the behaviors of Federated software. As long as everyone agrees to the meaning behind an Undo/Follow (“remove from collection”, which is a sensible assumption given we haven’t yet had incompatibility arising from misinterpreting ActivityStreams Activities), your app can do the necessary things required to make that an accurate reflection of it’s reality, and someone else in their software can do the same thing. If your software cares about groups for meetups, it can let an organizer know someone decided not to show up. If someone else’s microblog software just treats groups as a discussion, it can use that same interaction to notify its users that someone left the group chat.
Different “intents” between your software and the federated peer’s software. But each taken alone are entirely self-consistent, and taken together are still interoperable despite very different “views” of the Undo/Follow, even if the underlying ActivityStreams manipulation is the same.
You’re never going to be able to force federated software to do what you want, which is both the curse and blessing of such systems. So I wouldn’t optimize for attempting to do that.