Way to allow instantiating derived classes
You can do it however you like, really; however PHP will let you. 

Basically you need to make a decision at some point in custom_get_my_service() about which class you're going to use. Drupal's paradigm is hooks, so a contrived example might be:

function custom_get_my_service() {
  $service = &amp;drupal_static(__FUNCTION__, NULL);
  if (is_null($service)) {
    $service_class = custom_get_service_class();
    $service = new $service_class();
  }
  return $service;
}

function custom_get_service_class() {
  // Default
  $class = 'MyService';

  $modules = module_implements('custom_service_class');
  foreach ($modules as $module) {
    $function = "{$module}_custom_service_class";
    $class = $function();
  }

  return $class;
}


Then another module would implement the hook:

function another_module_custom_service_class() {
  return 'AnotherClass';
}


Obviously that's a very limited example, which will only let one module override the class, has race conditions, no error checking, etc. But that's the basic idea, I'm sure you can extrapolate from there based on what your module is providing.

You could also look at a DI approach, and have modules pass in their implementations of an interface you define into your functions, which then make the necessary use of them. It kind of depends on your level of experience with OO, and your opinions on 'best practice'.