Ajax form add library on callback
Is there any problem with the way I attach the library to AjaxResponse
that may cause the broken layout?

I guess so. This is how core does it when its an AJAX response, it attaches it to the form like this:
$form['#attached']['library'][] = 'my_module/library_name';
$ajax_response-&gt;setAttachments($form['#attached']);