Open ajax dialog with JavaScript
You can trigger Ajax in custom javascript. See How to Trigger Existing, Non-Form Ajax from JavaScript Event
In your case an Ajax dialog:
var ajaxSettings = {
  url: '/node/1',
  dialogType: 'modal',
  dialog: { width: 400 },
};
var myAjaxObject = Drupal.ajax(ajaxSettings);
myAjaxObject.execute();