How can I change the display settings of the referenced field, programatically?
One option is to alter the build array using entity_view.  You could then add conditional code there, to change image display settings.

/**
 * Implements hook_entity_view().
 *
 */
function mymodule_entity_view(array &amp;$build, EntityInterface $entity, EntityViewDisplayInterface $display, $view_mode) {
  if ($entity-&gt;getEntityType()-&gt;id() == 'paragraph' &amp;&amp; $entity-&gt;bundle() == 'my_paragraph_type') {
    if (!empty($build['field_image'][0])) {
      // $build['field_image'] = 
       // make changes to your image field here
    }
}