Administration page doesn't show pop-up (overlay) windows
The block layout is in the standard installation of drupal. This popup should work out of the box.

When you press the button "Place block" this ajax is used:

/admin/structure/block/library/mytheme?region=header&amp;_wrapper_format=drupal_modal...

So I would check first if this ajax is fired and second if there are any js errors in the browser debug tool (F12).