Can I use 'drupal_add_library' inside an ajax call?
drupal_add_library() under the hood ends up putting things into the same array as drupal_add_js() and drupal_add_css(). 

These get called by ajax_render(), so your function should work (so long as you are using ajax_render). 

However this will add the library load to every ajax request. This will slow them down some what, and most of the item you are using ajax for speed so that may not be such a good idea. Your html JS should be cached and it should only include the ui.dialog on pages where you need it. So I think you may be better off using the standard JS caching. 

In simple cases you may be able to use ajax_command_alert() in stead.