If neutral carries current back to the breaker panel, why doesn't it need to be connected to a switch?
The switch opens (turns off) the circuit, at which point electricity is not "flowing" through the circuit. The electricity flows to the first terminal on the switch, and then just sits there.  Think of it like a closed plumbing valve.  When the valve is closed, water flows to the valve, and then just sits there.  Once the valve is opened, the water can then flow through the plumbing.