Trigger an analog SPDT switch ONLY with a long button press
That'll be harder than you think.

As far as I remember, the gameboy buttons are scanned, i.e. the hardware activates a set of current sources in sequence, and for each of them checks which current sinks see a current flow.

That way, you don't need an pin for every single button. Compare reverse engineered schematics.

But that means that no button ever is "continuously" conducting current if pressed.

So, you'd have to build something that "counts" how many times within a time span a button has been closed whenever there was voltage across it; basically, do the same as the gameboy hardware does itself.

There's certainly clever ways of doing that, but let's be honest here: the by far easiest way, and with a solid chance also themmost compact one, would be a cheap small microcontroller hooked up to observe the voltage across the button matrix diodes (with high impedance, of course).

Doing so would have another benefit: Instead of then using an analog switch to change crystals (not even sure that works; analog switches have their own capacitances and phase behaviour, and your CPU + oscillator combo needs a specific loading capacity to work), you could unsolder the crystal, and replace it with a differential clock generated by your microcontroller; it's totally feasible to use the original oscillator (or a compatible one) to drive your microcontroller. That would then use a multiple of the oscillator's frequency (for example, 16× the oscillator's frequency) to generate its internal clock, and you could then generate any division of that clock with the built-in PWM unit that basically any microcontroller has.

Lower-tech solution: Use a reed switch / relay to switch out oscillators, and actuate that with an external magnet.