Heuristics for space-efficient storing of Unordered Finite Sets in a DFA
It sounds like you are describing a binary decision diagram, also known as a BDD.  BDD's have been studied extensively in the literature, so you might take a look at the literature.

In particular, the question of finding the best permutation is known  as the "variable ordering" problem for BDDs.  Finding the optimal variable ordering is NP-hard.  There are some heuristics, but they don't always work.  See https://en.wikipedia.org/wiki/Binary_decision_diagram#Variable_ordering.  Probably the easiest approach is to use an existing BDD library (the standard ones will typically incorporate some variable ordering heuristics).

You might also be interested in ZDDs.