Is it possible to create a custom theme function?
you don't have to use hooks in your theme functions.   you could create your own function called theme_table_custom()  and use that in your template files.

if for some reason you must use the hook, you could include the original drupal code in the function and include a check for a table id or something, which would then fire your custom code.