When is hook_file_download triggered
The easiest way to track where core hooks are called, from core code, is to look on the docs page.

For hook_file_download(), you'll see this:



A nice list of all implementations/invocations of the hook with links to the relevant functions.

For the cases that don't get covered there, I usually just grep the codebase for, e.g.

module_implements('file_download'


That takes me straight to file.inc, and the file_download() function, which I guess answers your main question. 

If that's not quite enough, I thoroughly recommend using Eclipse/Netbeans with XDebug which will let you set breakpoints to see what's happening in more depth.