Custom form in pop-up
There are two ways of doing this:


Using Modal Frame API
Using CTools


I've used both to successfully create modal popups with embedded forms. In both cases I first needed to create the form using Drupals form api and then assign an event for either MFA or ctools to bind to.

Overall I found MFA easier to get something up and running but ultimately ctools is more powerful and will give you a better understanding of a module that is now used widely throughout contrib modules.

Both modules documentation is the best place to start, MFA has great documentation pages and ctools has some excellent sample modules in the project itself.