Continuous controller to ternary switch
replace the pot with a 3 throw rotary switch.everything then becomes much simpler.
or you could try rthis.



simulate this circuit &ndash; Schematic created using CircuitLab
you'll need to adjust your pot's mounting so that the resistance can go down to zero and up to at-least 40K. it might need two diodes in series.