Private File System and User Access
I see two ways of doing that:


Change the "files-mode" to private via admin/settings/file-system (as googletorp stated) AND changing the filepath to somewhere outside of your webserved files.
Change the "files-mode" to private and set some rewrite rules inside your .htaccess (or elsewhere in your webserver config). this gives you more control which files you want served private and which public. Something like:

RewriteEngine On
RewriteCond %{REQUEST_URI} ^/sites/default/files/(private/.*)$
RewriteRule ^.*$ /system/files/%1 [L]



Now every request directly to the "private" files will be rewritten to the systempath which is controled by drupal. So no access without drupal is possible.