hook_theme() in a custom file of my module
Just to clarify, you mean the actual hook_theme() implementation, correct? Not a theme hook you are defining in hook_theme(). If so, by default, all hooks must be in the .module file. Hooks are meant to be callable at any time, and only .module files are loaded by default on every page load. However you can tell Drupal that a hook is in a different file by implementing hook_hook_info():

function hook_hook_info()
{
   $hooks['theme'] = ['group' =&gt; 'theme'];

   return $hooks;
}


With this code, Drupal will look for the hook (the array key 'theme') in the file MODULENAME.GROUPNAME.inc. In this case your GROUPNAME is defined as 'theme' in hook_hook_info(), so Drupal will load MODULENAME.theme.inc, whenever hook_theme() is invoked.

Now, if you meant that you want to put a theme hook that you define inside hook_theme() in another file, then you need to declare the file for that theme hook:

function hook_theme()
{
  return array
  (
    'some_theme_key' =&gt; array
    (
      'variables' =&gt; array(),
      'file' =&gt; 'path/to/file.inc', // relative to module root
    ),
  );
}


In this case, when the system tries to load the theme some_theme_key, it will load the file [MODULEFOLDER]/path/to/file.inc.