Show form errors before opening AJAX modal
I see two things that stand out to me:

1) Setting the '#name' attribute on the submit button. You should never do that. I've learned over the days that there is always a better way to do whatever you're trying to do when setting the #name attribute, and setting it causes some wonkiness. This may be why your form is not rebuilding.

2) You are rendering your status messages in a D7 way. Use this instead:

$messages = ['#type' => 'status_messages'];
$response->addCommand(new PrependCommand('#target-div', \Drupal::service('renderer')->renderRoot($messages)));
It looks weird, because the messages aren't being retrieved, but by setting the #type to status messages, Drupal will handle the retrieval in the background. This is probably why you are seeing some funkiness with the messages.