How to hash a set of elements
The basic problem seems to be that of canonicalization. If you can't reorder the set, then you will have to choose a commutative, associative, operator to concatenate the values; $a \otimes b = b \otimes a$ and also $a \otimes (b \otimes c) = (a \otimes b) \otimes c$

An immediate candidate for this would be to interpret the elements, $m_i$ as integers and let $\otimes$ be integer multiplication. You could then pass it through a secure hash function $H(\prod m_i)$ to get your final hash.

To improve the collision resistance with this choice of $\otimes$ operation (unfortunately $H(5 \times 4) = H(10 \times 2)$; different sets with the same hash) you might consider associating each message with a unique prime number, and by the fundamental theorem of arithmetic, this product will be unique.