process form in popup window
I would suggest you to have a look at Chaos tools suite; the module brings an example for a modal form inside a modal window right with it (AJAX example).
It should be not to hard to integrate that, if you're already working on a custom module.

All I've said is valid for Drupal 6; I haven't looked into cTools in Drupal 7.