How to not let users remove the entity reference once they added
Using Hook_form_FORM_ID_alter on the node form and disabling the field for users that are not the initial author should work. 

Code is untested but shows the general idea behind the solution

HOOK_form_FORM_ID_alter($form, &amp;$form_state){
  global $user; // Get the current logged in user information
  if($user-&gt;uid != $form_state['node']-&gt;uid){ // Is the current user not the original author of this node? 
    $form['FIELD_REFERENCE']['und'][0]['#access'] = 0; // Restrict access to this field (You'll have to check where exactly in the tree to place this).
  }
}