How to ensure that only unique row combinations are present? I have three tables which can be called set, item and assignment. The assignment table assigns an item to a set, thereby my goal is to ensure that each combination of items is unique. Hence, if item A and item B form set 1, there is no second set allowed which is formed by item A and item B. A combination of item A, item B and item C, however, is distinct and hence ok.

My original approach was using a hash with a unique constraint for each set which is formed by the IDs of the associated items. This hash would then be updated by a foreign table trigger on each insert/update operation on the assignment table. However, I failed in implementing such an update function and don't even know whether it's possible or not.

Does anybody know whether this is the way to go or whether there's an easier solution to my problem?
Triggers will not work on some scenarios. Let set1(A,B), set2(A,B,C) Now one cann't move to set2(A,B,D) by dropping set2-C and adding set2-D assignments because DB must go through prohibited set1(A,B), set2(A,B) state.
You should implement custom API, stored procedures or something with proper transactions which will check only final target state. Also those procs should place locks on updated set rows during the transaction. Because really what is updated is set.