Role level file access control
The first step is using private files, because you can not control access to to public files as Drupal is not involved when they are delivered.

Private files in plain file fields (file and image, not media entity) automatically respect node access and will only be accessible if the user has access to at least one node that is using the file.

As you mention that users can bypass node access, I assume you have node access already implemented and it should then just work