How to figure which function(s) are triggered upon file download to check permissions I am experimenting with private file system, done custom way :) .

There is file system/files/test.txt (That is, private://test.txt).

When I try to access the file, I get access denied from Drupal, as expected (site redirects to Access denied page, and dblog shows that accesss have been denied).

But now I want to know what module/function have blocked access to the file. I am blindly trying to check hook_file_download or hook_file_download_access, but as far as I can see it does not go through that functions.

All that in attempt to solve this.
If the Drupal log shows you that the access has been denied, then you have a simple way to debug this: if you have a debugger, simply set a breakpoint in drupal_set_message. Skip calls to that function (there shouldn't be too many) until you get the one with the access denied message - and then look at the backtrace. This should show you which function denied the access.

If you don't have a debugger, you can simply (temporarily) add some extra code in that function to test for the relevant error message, and then output a backtrace using debug_backtrace. You can output the data on the screen, to a log file using drupal_debug (if you have the devel module enabled), add it to the generated error message, etc.