Change a normal submit 'input' type to 'button' type with button tag
Well that was quick... I used my intuition and was able to accomplish it by doing the following:

In mytheme.theme:

/**
 * @param $suggestions
 * @param array $variables
 */
function mytheme_theme_suggestions_input_alter(&amp;$suggestions, array $variables) {
  $element = $variables['element'];

  if (isset($element['#attributes']['data-twig-suggestion'])) {
    $suggestions[] = 'input__' . $element['#type'] . '__' . $element['#attributes']['data-twig-suggestion'];
  }
}

/**
 * @param $form
 * @param \Drupal\Core\Form\FormStateInterface $form_state
 * @param $form_id
 */
function mytheme_form_alter(&amp;$form, FormStateInterface $form_state, $form_id) {
  if ($form['#id'] == 'views-exposed-form-search-results') {  
    $form['actions']['submit']['#attributes']['data-twig-suggestion'] = 'search_results_submit';
    $form['actions']['submit']['#attributes']['class'][] = 'search-box__button';
  }
}


In input--submit--search-results-submit.html.twig

{#
/**
 * @file
 * Theme override for an 'input' #type form element.
 *
 * Available variables:
 * - attributes: A list of HTML attributes for the input element.
 * - children: Optional additional rendered elements.
 *
 * @see template_preprocess_input()
 */
#}
&lt;button{{ attributes }}&gt;
  &lt;svg class="icon search-box__open"&gt;
    &lt;use xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="/themes/custom/mytheme/build/img/svg-sprite.svg#icon-search"&gt;&lt;/use&gt;
  &lt;/svg&gt;
&lt;/button&gt;


Form still submits and functions as expected. Is this the general approach or at least acceptable? I will go with this unless there is a better way.