Hook to log results of calls to drupal_mail() made by other modules?
The only way I can see is implementing hook_watchdog(), and look for a error message that matches "Error sending e-mail (from %from to %to)." 

function mymodule_watchdog(array $log_entry) {
  if ($log_entry['message'] == 'Error sending e-mail (from %from to %to)') {
    // ...  
  }
}


The alternative would be implementing a class that replaces DefaultMailSystem, which is a class implementing MailSystemInterface. To achieve this, it is enough to change the value of the mail_system Drupal variable.