Pull down through LED
Although LEDs are mainly used to generate light based on an electrical stimulus, the reciprocal is also true.  Referring to the original schematic, when the switch is open, the anode of the LED is connected to a high impedance input.  When the LED is exposed to light, it can generate a voltage across it.  Not sure if this is what you are seeing, but certainly can be a potential issue.

As mentioned, adding a resistor in parallel with the LED also gives a path to ground to shunt current from the LED when switch is open, but coming up with a value that works under all conditions may be difficult since there are unknowns.  To make it full proof, you can add a buffer between the logic gate input and the LED.