Removing file from form breaks site
The value of $form_state['values']['csv_file'] is empty (0) when you remove the file. You need to do a sanity check on $file after file_load not only for that reason but also as a good coding practice.

function custom_admin_form_submit(&amp;$form, &amp;$form_state) {
  // Saving the file, setting it to a permanent state, setting a FID variable
  $file = file_load($form_state['values']['csv_file']);
  if ($file) {
    $file-&gt;status = FILE_STATUS_PERMANENT;
    file_save($file);
    variable_set('csv_fid', $file-&gt;fid);
  }
  else {
    variable_set('csv_fid', '');
  }
}