how do I open a custom module form in colorbox popup?
I am suggesting you to use chaos tools for modal popup form. Its easy to implement and integrate into Drupal 7. I already used in my several sits.

You can find a great example of Ctools modal form at here