Flip Switch with transistor and push button
Your idea 'current follows the path of least resistance' is an approximation, and it holds only when there is a limited amount of current. In your case you use a lab power supply, which is a constant voltage source, which will supply all the current you ask from it. Hence both the transistor and the red LED parts get current.

A way to switch the other LED of when you press the switch is show below.







simulate this circuit &ndash; Schematic created using CircuitLab

The principle is that with the switch open, Q1 gets a base current via R1 and D1, so it conducts, and current flows through R2, D2, Q1, so D2 lights up. But the base current is too small for D1 to light up.

When you close the switch, Q1 gets no base current, and the current through D1 is large enough to light it up.