Pulldown doesn't seem to affect inverter?
If you are actually using an original, old, 7406, or a 74LS06, the 10K resistor is much too high to produce a logic Low.  Bipolar TTL inputs source current, so a pull-down resistor must be a low enough value to draw about 1 mA when dropping less than 0.8 volts to produce a Low.  Traditionally, we would use a switch to ground to guarantee a Low, and a 5K or so pull-up resistor to guarantee a high, although an open input would normally be recognized as a High.

A CMOS part (74HC06, 74AC06, etc.) has a very high input impedance and neither sources or sinks any significant current so would work in your circuit.