How to open a popup form from a view link?
Your code is building an ajax response. This only works in ajax callbacks, nowhere else.

If you want to make a link modal, so that it will open in a popup, you have to set the class use-ajax and the attribute data-dialog-type. Like in this example from ConfigSync.php:

$links['view_diff'] = [
  'title' =&gt; $this-&gt;t('View differences'),
  'url' =&gt; Url::fromRoute($route_name, $route_options),
  'attributes' =&gt; [
    'class' =&gt; ['use-ajax'],
    'data-dialog-type' =&gt; 'modal',
    'data-dialog-options' =&gt; json_encode([
      'width' =&gt; 700,
    ]),
  ],
];


For modal links to work you need to attach a library, see How to create modal windows (pop-ups)?