Why can't I call theme functions declared in template.php in mymodule_nodeapi hook?
Functions defined inside of template.php will not reliably be available (nor are they meant to be) outside of the theme. When it comes to overriding preexisting theme functions, the overrides created within template.php will automatically be applied by Drupal core, there is no reason to actually invoke the function. 

For example, if you wanted to override the functionality of theme_menu(), and you created in your theme's template.php a function named mytheme_menu(), that new function will not actually be invoked anywhere within the site's code. The function that is invoked will still be theme_menu() and Drupal core will automatically override that invocation with the functionality within mytheme_menu(). This is the intended use of the template.php file; to override existing themable output, not to create new theme functions.

Now, if you create a custom element and need to create a brand new theme function for that custom element, this function must be declared within the context of a custom module. For example, you would need to define the function theme_myelement() inside of mymodule.module. You will also need to implement hook_theme() by defining the function mymodule_theme() in order to tell the Drupal theme system about your new function. There's a bit more to it, which is explained in the Drupal module developer's handbook for Drupal 6 and Drupal 7.

If this doesn't answer your question, please edit your question to provide specific details about your custom module and custom theme functions.