Why is this binary state format so much more efficient than a non-binary one?
You are increasing the state space by introducing non-binary options. This increases the complexity the model has to learn to reach the same level of performance.
In particular, you are creating a nested structure which makes it more difficult to assign reward credit to a specific option. The hierarchical options are &quot;Should the snake move?&quot; then &quot;If yes, which direction?&quot; make it slower to assign credit to the directional movement. A flat structure &quot;Should the the snake move up?&quot; allows credit to be more directly assigned to that decision.