User group / user access to files Drupal 7
Drupal supports this natively.

Use private file access and content_access to support granular access to your content.

Due to some bugs here and there I had to rebuild the permissions (/admin/reports/status/rebuild) after configuring content_access a usual clear cache wont work here.