Oscillator slows down when my hand hovers over it
While following the clock trace in the layout (thanks @winny to point me in the right direction) I realized that the oscillator was connected to pin 12 of the chip, not pin 62.

The reason I was focusing on pin 62 is that I did a continuity check between the X1 pins and the CPLD. I was probably fooled by a common ground connection. I should have carefully followed X1 traces and found the 33 ohm resistor in series, then followed the trace up to pin 12.

And I suppose the reason pin 62 was still kinda working is because it was a floating pin picking up some of the signal from the nearby clock input.

Of course none of this could have happened if I had a board schematic or some other documentation, but the website (www.ourfpga.com) seems to be dead.