How to create modal windows (pop-ups)?
This the official documentation for the API change you've mentioned:

Modal/dialog/ajax is using query parameters instead of accept headers

The most important detail is to attach this library:

$build['#attached']['library'][] = 'core/drupal.dialog.ajax';


You ask about why this works on admin pages without this. The reason is, admin pages already have dependencies on most Drupal jQuery libraries, while non admin pages are loaded without any jQuery out of the box (which is a great performance improvement in D8).

Bootstrap Theme

If enabled in the theme settings of the Bootstrap theme, "jQuery Modal" is replaced by 
"Bootstrap Modal", see this code snippet:

LibraryInfo::alter()

elseif ($extension === 'core') {
  // Replace core dialog/jQuery UI implementations with Bootstrap Modals.
  if ($this-&gt;theme-&gt;getSetting('modal_enabled')) {
    $libraries['drupal.dialog']['override'] = 'bootstrap/drupal.dialog';
    $libraries['drupal.dialog.ajax']['override'] = 'bootstrap/drupal.dialog.ajax';
  }


For this to work you attach the same core library as above, so that the Bootstrap theme can find these core libraries to override them.