Remove entities programmatically from view results before pager is generated
You need to alter the query in one way or another, that is the only way the pager can work.

Sounds like you are using nodes, then you can use the node access grant system. Basically, it allows you to "index" the access checks and assign attributes to the nodes, like the group they belong to. And then there is a second hook, that allows you to define what groups (or whatever it is in your case) the user has access to. This will then be combined into a query for you.

Be aware that this is one of the most complicated systems in Drupal, but if you can't do the access check directly as part of the query (like checking the value of a certain field), then this is the only way.