How do I open a custom module ajax form in colorbox popup?
If you output your form hidden in the html, you can pop it up with colorbox_inline.

If your form lives in a menu callback and has a path, you can put a rendered link to your page everywhere and it will open modal with colorbox_load - see also ng_lightbox, if you put your path in it's config.