What theme function outputs the administrative overlay in D7?
To output the overlay, Drupal uses a template file, overlay.tpl.php. To use that template file, the Overlay module implements hook_page_alter(), which contains the following code:

  $mode = overlay_get_mode();
  if ($mode == 'child') {
    // Add the overlay wrapper before the html wrapper.
    array_unshift($page['#theme_wrappers'], 'overlay');
  }


The Overlay module has also a preprocess function for that template file: template_preprocess_overlay().