How are access permissions prioritized when multiple modules implement them? I have the Group module setup where I only allow view permission for group content to group members, so I have closed groups. But additionally, I want groups to also make some content public to non-members (but authenticated users) or to anonymous users even. 

I have setup the Permissions by Term module for this where I made an "Additional view permissions" vocabulary that has two terms "all authenticated users" and "public". Those two terms I have granted permission to the respective roles. I had hoped that this would override the group permissions, but it doesn't. If I am not logged in, group content that has the "Additional view permissions" set to "public", still is not accessible to anonymous users.

I guess my question is: how does Drupal determine the priority of multiple modules arranging permissions for the same node, and can I influence this? In my case the group permissions seem to take priority, causing view permissions to be denied. But if it would first look at the permissions by term module, it would actually allow access.
Have a look at this feature request Allow other modules to alter gnode_node_access's forbidden result, especially this part of it:


  If the group node's gnode_node_access module doesn't find a reason to return neutral, or to grant access, it returns a AccessResult::forbidden(). This prevents any other module from granting access. ...


So to "determine the priority of multiple modules arranging permissions for the same node" (as in your question): if the Group module denies access, then there is no way for any other module to overrule that access.

Maybe not what you like to hear/read, but this is how access is handled via Group module.