How to restore private file structure after major version update? After a Drupal 6 to 7 update we are unable to use private image download, symptom is we cannot access private files via system/files, while public files can be loaded.

Trying to download private file results in a 404 -- Not found error, sent from the web-server directly, not drupal. After some changes to the configuration this results instead in a Drupal Error page. The server log says 

File does not exist: /home/licebase/drupal/system/files/genefinding_diagram_1_2.png, referer: https://myhost/node/761470/edit?render=overlay

The system file exists and has the correct permissions.

Centos 6 on virtualization host, PHP 5.3.3, Postgres 8.4 on dedicated DB server, Apache 2.2.15 (centos) + a lot of active modules + fields+cck
Edit: We have a workaround, I hope this question can serve as a good practice guide of good to debug complex software like Drupal showing unexplainable errors or WSOD. Wish we had followed our own advice to not waste time on our futile attempts time.

tl;dr It was the core module Database logging 7.42    Logs and records system events to the database. which is interfering with private files in our specific setting. We disabled most modules including core, and brought them back one-by one. Database logging is reproducibly inducing the error on activation.


Don't panic and make a debugging plan, listing options of what could be checked, and then stick to it until everything has been checked. If out of options, try to shake things up
Compare your system to a vanilla system and note down what is does differently
Modules might interfere with each other in unexpected ways or with your specific configuration. That doesn't mean there is a bug in any of those modules. Boil down your system to the least complex configuration possible, remove any unneeded entry in your configuration file, disable any module that can be disabled including core modules, suspect everyone. 
Check if error still present, if not re-enable all modules one-by-one, and note down which configuration change causes the error. Otherwise, start debugging going to 1.


Cheers