Performance for file serving and storage
If you are storing them the public file system, Drupal does not really interact with them other than to provide the URL.  The .htaccess rules check if the request is for a real file.  If it is a real file, the web server service it up.

So for individual files, it is more about can your webserver handle the load.  You probably want to look at using a static file caching service in front of your site.  CloudFlare does a good job and its basically free. Some alternates are to have a varnish server or a caching Nginx server in front of your server.

Drupal has been used for a lot of sites with a lot of users, files and the like.  E.g. I know that Story Corps uses it internally to manage the thousands (millions) of interviews  they have.

Just be aware that if you need to have a large volume site, you will need to plan for a large volume system setup.  This is true of ANY web software you use.