How to add a secondary Case Role that has access to Activities they are not 'on', eg ones that existed before they were added
I'd say this is coming from CRM_Case_BAO_Case::checkPermission(), around line 2659 where it says

//user has only access to my cases and activity. [sic]
//here object specific permmions come in picture.


It seems like this is part of the definition of "my cases and activities", where you have to be either the reporter, with contact/target, or assignee of the activity to be able to view it.

So to get the arrangement you're looking for you either have to give them access to all cases and activities, or something more complicated involving code is needed.