How can I register a image_style theme function in my module which only works in my theme's template?
You need to use hook_theme_registry_alter() to replace the callback function for the image_style theme. Something like this should do it:

function MYMODULE_theme_registry_alter(&amp;$registry) {
  $registry['image_style']['function'] = 'mymodule_image_style';
}