Does this design smell bad? Can I pull a pin simultaneously down and up?
Yes, it smells. Use a transistor.  This is the normal way to invert a signal.  The OE input is a sourcing input.  These are designed to be used with a switch which is either open or switched to common.  It's not meant to have Vcc connected directly to the input!  You have a switch (your header), which would be perfect, but you want the behaviour inverted - so invert it with a transistor.  That's why it's called TTL. The resistor to ground also increases the noise sensitivity of the circuit needlessly.



Here when the header is open the base of the transistor is high and OE is pulled low.  When the header shorts the transistor turns off and OE goes high via the internal pullup.  I've shown 10k to the transistor base here, which is a bit greedy for power - a lot of values would work here, though.  The higher you go the less power it draws, but the more sensitive to noise you become.

If you can spare the current and want a more noise resistant circuit you can also tie OE to Vcc with a parallel pullup (Rp) to the internal 270k.  Absent a compelling reason not to, giving the input a lower pullup resistor here is probably a good idea.