How can I tell if a shortcut is already used?
This isn't possible since the app doesn't store what shortcuts it may make available, as it can change the menus as it wishes or use a shortcut for something not available in a menu.


When user shortcuts are added and the user presses the shortcut, macOS looks through the menus to find the menu item which matches the name you enter. This is a runtime check performed when the shortcut is actioned, since it's not possible to check prior to this. This is why the text which you enter for the menu item in System Preferences is a freeform box, since it's not possible to enumerate every menu item which could exist for an app.
Apps don't have to restrict shortcuts to items available in the menus. Apps can trigger on shortcuts without registering a companion menu item. This is the second reason the shortcuts can't be enumerated.