Incorrect XTAL waveform from PCB
You should be using ground pours. Using long traces for ground, especially on your crystal, will introduce a lot of noise. I'm betting that's a large part of your problem.

See the following app note for PCB routing of a crystal oscillator:

http://ww1.microchip.com/downloads/en/DeviceDoc/Atmel-8128-Best-Practices-for-the-PCB-Layout-of-Oscillators_ApplicationNote_AVR186.pdf

Internal planes are highly preferred in designs such as this. These allow you to drop a couple of vias straight down to ground through a relatively low-impedance connection. This will minimize any noise and ensure your oscillator behaves as expected. If you absolutely cannot use internal layers (you are restricted to a 2-layer board) then basic polygon ground pours will suffice, but you must ensure they have low-impedance paths to the supply. If your pour necks down in areas or has long, roundabout paths to get back to the supply, then you'll still have issues with noise.