Enforce uniqueness of custom field
I think a hook's the way to go.  For non-custom fields you can specify this a little more easily in the XML definition of the field, but for a custom field, I think the hook is your best best.

I think I'd use hook_validateForm or hook_pre.  The first is more user-friendly, but since it's at the form level you have to identify everywhere the field can be edited.  It's a fairly easy hook to write though - just make an API "get" call to CustomValue with that value and see if you get a count > 0.