How to do a Library Override in my theme for dialog.ajax.js
Looks like you're just missing the core/ prefix:
libraries-override:
  core/drupal.dialog.ajax:
    js:
      assets/js/dialog.ajax.js: false