Admin custom module not displaying in theoverlay
There is Overlay Paths


  This is a simple module that enables site builders to use the built-in
  Overlay for non-admin pages.


Refer 

Working with the overlay

https://drupal.stackexchange.com/a/8265/4471