Form fields created via Ajax are missing element attributes
It seems like a good idea to invoke the callback in the form build to include the form elements of the callback in $form to allow the form builder to process these elements. But then you overwrite the same form elements that the form builder has processed with the unprocessed form elements when the ajax triggers the callback.

Solution: Move all form elements in buildForm(). Leave the callback empty, only return the wrapper part of the form.