How to open file in new tab instead of download when submitting form?
As @clive points out, this cannot be done on the server side. Check this post for a javascript based solution. You need to attach the javascript to your form during build. See below for an example of form build with a javascript attached to the submit element:

/**
 * {@inheritdoc}
 */
public function buildForm(array $form, FormStateInterface $form_state) {
  // Assuming you are extending core \Drupal\Core\Form\FormBase class
  // and prior the lines below you built your form elements.
  $form['actions']['#type'] = 'actions';
  $form['actions']['submit'] = [
    '#type' =&gt; 'submit',
    '#value' =&gt; $this-&gt;t('Download file'),
    '#button_type' =&gt; 'primary',
    '#attached' =&gt; [
      'library' =&gt; 'your_module/submitToNewTab'
    ]
  ];

  return $form;
}