Drupal 7 - Ajax form in modal window
It's hard to tell what you doing from your description. A better description could possibly yield better answers.

One possible problem could be that the form needs to be loaded on the page the it is submitted on. This is required for the validate and submit handlers to be processed.