How to use hook_url_outbound_alter for emails only?
Can I use hook_url_outbound_alter() to only alter URLs when it was invoked from a specific module or something?


You cannot, if that module doesn't put in $options (the same argument passed to url()) something that makes you understand that. To make an example, entity_uri() uses $options['entity_type'] and $options['entity']; if you were trying to intercept the calls to url() done from entity_uri(), you could check the presence of those options.
Otherwise, the context given to hook_url_outbound_alter() doesn't allow to understand which module called url(). Maybe you could get that using debugging functions, but I am not sure I would put that kind of code in a module.

Since you seem to trying to alter links in an email sent from Drupal, you could implement hook_mail_alter(), assuming the emails are sent using the correct way of sending them with Drupal.