How to load a library only once, or how to check if a library has been loaded?
EDIT: Sorry didn't see you were doing it via AJAX, my below example might not be the same. But the theory of the JS triggering multiple times rather than the actual files loading might still be valid

I don't think it's the piece of code you've showed is causing your issue. Is the  library files definitely getting loaded multiple times into the source code? Or is it that the effects of the library are happening multiple times? (e.g. a JS being trgiggered multiple times) as thats a different issue

Drupal handles this situation and when it aggregates all the required libraries so any duplicates are removed. For example I'm just working on a news listing page that has 10 nodes coming out and just replicated your code to print "hi" add a library and the "hi" comes out 10 times but the library files only once