Do I need buffer inverter for a Crystal Oscillator?
You can use what you have shown but personally I'd make a 2:1 logic multiplexer from 4 NAND gates: -



There are plenty of other options courtesy of google images.