How can I listen for drupalOverlayBeforeClose in custom jQuery?
You need to use the following code,

jQuery(document).bind('drupalOverlayBeforeClose', function(event) {
   // your code
   return false // This will prevent the overlay from closing.
});


click, dbclick and other methods are shortcut methods for bind('click', function(){}), bind('dbclick', function(){}) respectively. However there is no shortcut method called drupalOverlayBeforeClose, that's why the following code didn't worked,

$(document).drupalOverlayBeforeClose(function(event) {

});


So you need to bind drupalOverlayBeforeClose explicitly using bind. If you return false in your callback, then the overlay won't be closed.

Hope this clarifies the answer.