Assign User Permissions to a File?
The closest out-of-the-box option is to use a combination of the public (public://) and private (private://) file system.  Public files can be access by anyone with a url to the file. They're served directly from the server without really having to deal with drupal.  Private files on the other hand require the user to log in.

The downside to the private file system is the files are basically streamed to the end user via php which is slower. There are ways around this but if you're not expecting heavy download traffic then I wouldn't worry about it just yet.

The way access control is handled is via the node.  You can create a single filefield and attach it to a any content type.  Let's just say we're using the 'article' content type for now with a single filefield that supports both public and private storage.  When you add a file to the article you can choose the private file system.  You can then use any of the node access control modules (tac_lite, og, etc.) to control who can see the node.  Those that can see the node can then download the file.