How to fetch/load a node form using jquery/ajax?
I don't think you've got the right parameters on the jQuery "load" method. The jQuery API page has this for the signature:

.load( url [, data ] [, complete ] )


This might be a working revision (not tested):

$('.custom-form').load('?q=node/add/custom', {}, function() {
     Drupal.attachBehaviors($('.custom-form'));
});


However, your strategy will load the entire Drupal page, not just the form. The AJAX-loaded page and form will not be in the expected request context - I'm not sure that the form can be successfully submitted in this circumstance.

You might try looking into implementing CTools Modal API instead, which is designed to allow the loading forms into modal windows via AJAX.