How to clear unused files from the database and the filesystem?
Drupal should be taking care of this for you. The pattern that modules should be following is: 


When you're adding a new file call file_save() (or a function that calls calls it) then call file_usage_add() to let the system know it's being used. 
When you're done with a file call file_usage_delete() to release it then call file_delete($file, FALSE). The FALSE indicates that it should be deleted only if nothing else is using it. 


So unless you've got a buggy module there probably aren't to many unused files sitting around but if you wanted to check it you could run the following query:

select fm.* from file_managed fm 
left join file_usage fu on fm.fid = fu.fid 
where fu.fid is null


I ran that on a test site and noticed that the devel_generate module isn't properly registering usage of the user's pictures when it creates them. You might notice similar things.