op amp as comparator/switch - strange behaviour
What you describe SHOULD work, so it seems you may be just crossing over a limiting boundary when you change modes. Try it with NO load on the op-amp. If that still doesn't work then circuit is not as you believe OR opamp is damaged. 

LM321 data sheet

Presumably the chip enable pin does not provide a massive pullup? Or does it?
Try it with nothing connectd to LM321 output pin.

Things to look at include:

Supply voltage. 3V in datasheet so 4V2 or 5V OK.

Max Vcommon_mode = Vdd - 1.5V or 2V depending on conditions. In either case you SHOULD be well below that.

Massive output pullup - see above. The LM321 and LM314 (quad big brother) data sheets provide sink currents at high supply voltages but not at low voltage. IF these decrease with supply voltage you may be below the 1 to 2 mA needed by the LED etc.