Restrict media files to other users once uploaded
If you have Groups on your site with grouped access permissions, I would recommend using the Group module for less complex group permissions like this and OG for configuring more involved group permissions.

Using the Group module you will be able to section off the media and other content like this.

EDIT:
I suggest you configure Group functionality / permissions directly on the File / media entity, rather than just the nodes they are added / used from. Failing that, I believe you will have to (clone and) override the view that the media entity browser uses and do some filtering based on group(s) of the media file author.

Then try using the user id of the author initially using static values to test your view. If the UI of the view does not provide what you are looking for, you might need to write your own Views Access Plugin. Here is one I wrote for Message Private module: InboxPermission.php. You could do similar.