What is the purpose of rerender FormState property?
The only place in core that checks for, $form_state-&gt;get('rerender') in an if statement is in:
/core/modules/views_ui/src/Form/Ajax/ ViewsFormBase.php
// If the form has not been submitted, or was not set for rerendering, stop.
if (!$form_state-&gt;isSubmitted() || $form_state-&gt;get('rerender')) {
  return $response;
}

// Sometimes we need to re-generate the form for multi-step type operations.
if (!empty($view-&gt;stack)) {
  $stack = $view-&gt;stack;
  $top = array_shift($stack);

  // Build the new form state for the next form in the stack.
  $reflection = new \ReflectionClass($view::$forms[$top[1]]);
  /** @var $form_state \Drupal\Core\Form\FormStateInterface */
  $form_state = $reflection-&gt;newInstanceArgs(array_slice($top, 3, 2))-&gt;getFormState($view, $top[2], $form_state-&gt;get('ajax'));
  $form_class = get_class($form_state-&gt;getFormObject());

  $form_state-&gt;setUserInput([]);
  $form_url = views_ui_build_form_url($form_state);
  if (!$form_state-&gt;get('ajax')) {
    return new RedirectResponse($form_url-&gt;setAbsolute()-&gt;toString());
  }
  $form_state-&gt;set('url', $form_url);
  $response = $this-&gt;ajaxFormWrapper($form_class, $form_state);
}
elseif (!$form_state-&gt;get('ajax')) {
  // if nothing on the stack, non-js forms just go back to the main view editor.
  $display_id = $form_state-&gt;get('display_id');
  return new RedirectResponse(Url::fromRoute('entity.view.edit_display_form', ['view' =&gt; $view-&gt;id(), 'display_id' =&gt; $display_id], ['absolute' =&gt; TRUE])-&gt;toString());
}
else {
  $response = new AjaxResponse();
  $response-&gt;addCommand(new CloseModalDialogCommand());
  $response-&gt;addCommand(new ShowButtonsCommand(!empty($view-&gt;changed)));
  $response-&gt;addCommand(new TriggerPreviewCommand());
  if ($page_title = $form_state-&gt;get('page_title')) {
    $response-&gt;addCommand(new ReplaceTitleCommand($page_title));
  }
}

The if (!empty($view-&gt;stack)) { part will only pass if the form is a multi-step form and you're not in the last step; otherwise, it will proceed to close the modal. Ex: when you click &quot;Apply&quot; or &quot;Cancel&quot; or &quot;Remove&quot;. (Ex: when adding a field) it triggers the else:
else {
  $response = new AjaxResponse();
  $response-&gt;addCommand(new CloseModalDialogCommand());
  $response-&gt;addCommand(new ShowButtonsCommand(!empty($view-&gt;changed)));
  $response-&gt;addCommand(new TriggerPreviewCommand());
  if ($page_title = $form_state-&gt;get('page_title')) {
    $response-&gt;addCommand(new ReplaceTitleCommand($page_title));
  }
}

Conclusion
As it only gets checked in the views_ui module, this setting is basically a way to tell the Views UI Form:
if the setting form is a multi-step form, like when you're adding a field:

to keep the modal open and build a new $form_state for the next form

Otherwise, its like when you click apply/cancel/remove it tells it to close the modal, show save &amp; cancel buttons, and show/refresh the live preview.