Update entity ref field in parent form with new node created in ctools modal
There are few existing modules which gives the same functionality.
 - https://www.drupal.org/project/entityconnect
 - https://www.drupal.org/project/references

If you want to handle it in custom way, fetch the field and set the default (target_id) value in the form array element and use drupal_render/drupal_render_children to convert the form element array into html and try to replace it in the front end.