How to render ctools_custom_content_type_edit_form with ctools_modal_form_wrapper()
Look at if ($settings['custom_type'] == 'fixed') condition in ctools_custom_content_type_edit_form(). Problem in it I think. Also, look at ctools_custom_content_type_get_conf() and do all needed to this function return $settings['custom_type'] equal to 'fixed'.
Maybe this will fix problem, change this value before ctools_modal_form_wrapper() like this: $form_state['subtype']['name'] = 'custom';