How do you use the Drupal 7 overlay on the node/edit screens for your popups when it's already being used?
It is possible to change the z-index used by the overlay, but I don't think it's necessary. You just use for your overlay a z-index that is higher than the one used by the Overlay module.

In the case you need to know when the overlay is output, or how it is output, you should check the code executed by overlay_page_alter(), which is the implementation of hook_page_alter() made by overlay.module.

function overlay_page_alter(&amp;$page) {
  // If we are limiting rendering to a subset of page regions, deny access to
  // all other regions so that they will not be processed.
  if ($regions_to_render = overlay_get_regions_to_render()) {
    $skipped_regions = array_diff(element_children($page), $regions_to_render);
    foreach ($skipped_regions as $skipped_region) {
      $page[$skipped_region]['#access'] = FALSE;
    }
  }

  $mode = overlay_get_mode();
  if ($mode == 'child') {
    // Add the overlay wrapper before the html wrapper.
    array_unshift($page['#theme_wrappers'], 'overlay');
  }
  elseif ($mode == 'parent' &amp;&amp; ($message = overlay_disable_message())) {
    $page['page_top']['disable_overlay'] = $message;
  }
}