How can I restrict that only sender and receiver can download the attached file with the private message?
Your question is fully answered here http://drupal.org/documentation/modules/file

Please read sections:


Managing file locations and access
Accessing Private Files



  You can also specify whether files are stored in a public directory or in a private file storage area. Files in the public directory can be accessed directly through the web server; when public files are listed, direct links to the files are used and anyone who knows a file's URL can download the file. Files in the private directory are not accessible directly through the web server; when private files are listed, the links are Drupal path requests. This adds to server load and download time, since Drupal must resolve the path for each file download request, but allows for access restrictions.