How can I restrict that only sender and receiver can download the attached file with the private message? I am using drupal 7. I created a new file field in private messages for attachments and I want that only sender and receiver should be able to download the file attachment. 

I apologize if this question is duplicate but I didn't find the answer for this.

EDIT: Just one addition. I am using Custom Field permissions module as well. And authenticated users are unable to see the attachment until "View anyone's value for field" is enabled for them.
Your question is fully answered here http://drupal.org/documentation/modules/file

Please read sections:


Managing file locations and access
Accessing Private Files



  You can also specify whether files are stored in a public directory or in a private file storage area. Files in the public directory can be accessed directly through the web server; when public files are listed, direct links to the files are used and anyone who knows a file's URL can download the file. Files in the private directory are not accessible directly through the web server; when private files are listed, the links are Drupal path requests. This adds to server load and download time, since Drupal must resolve the path for each file download request, but allows for access restrictions.