Finding all possible reflection matrices for a given Wythoff construction
I found an answer in the form of the Todd-Coxeter Algorithm from group theory. This still involves iterating over permutations of the group, and removing duplicates. However, the group can be defined using symbols to represent different sequences of operations, and duplicate-matching performed on these symbols rather than elements of the group itself.

For example, if a pair of mirrors A and B have an angle of (PI/3) then they form a cycle of order 3 (since a reflection through a pair of mirrors equals a rotation through twice the angle between them). So the sequence ABABAB maps any element back onto itself. By constructing a path like this for each pair of mirrors, it's possible to define new elements for each step along a path, then eliminate duplicates either through string-matching operations or by constructing a graph and eliminating equivalent nodes. For large sets, this should be much more efficient than multiplying and comparing matrices directly at each step.