Programmatically select image style for use with custom file entity view mode
Sounds like something like this should work for you, to set the file display for the image file type, view mode name VIEW_MODE (replace with your custom view mode), to display original image. It either adds a new file display or updates an existing one:

$displays = file_displays_load('image', 'VIEW_MODE', TRUE);
$formatter = 'file_field_image';
if (array_key_exists($formatter, $displays)) {
  $display = $displays[$formatter];
}
else {
  $display = (object) array(
    'api_version' =&gt; 1,
    'name' =&gt; 'image__VIEW_MODE__' . $formatter,
  );
}
$display-&gt;status = 1;
$display-&gt;settings['image_style'] = '';
file_display_save($display);