Can any function be used as a hook? I'm looking to override format_size(), but I'm not sure if this is possible in a module.
If I implement myModule_format_size() or in a theme myTheme_format_size() will the hook be called?
No.
Not all functions invokes a hook in other modules. As we can see in http://api.drupal.org/api/drupal/includes%21common.inc/function/format_size/7 there is no module_invoke_all is called.

Here is how we define module_invoke_all  


  Invokes a hook in all enabled modules that implement it.


function module_invoke_all($hook) {
  $args = func_get_args();
  // Remove $hook from the arguments.
  unset($args[0]);
  $return = array();
  foreach (module_implements($hook) as $module) {
    $function = $module . '_' . $hook;
    if (function_exists($function)) {
      $result = call_user_func_array($function, $args);
      if (isset($result) &amp;&amp; is_array($result)) {
        $return = array_merge_recursive($return, $result);
      }
      elseif (isset($result)) {
        $return[] = $result;
      }
    }
  }

  return $return;
}