Jquery popup stops the ajax in the form fields
Maybe not the answer you're looking for, but things would probably work if you load the form page within a iframe. I've used this approach successfully with forms that have AJAX fields. In D7 the admin overlay is also built using iframes.