how to open custom form in popup onclick in menu link drupal
There are many ways to do this. 
You can use  Popup forms module. Its configuration is in documentation of module 
or Modal Frame API module.