Oscillator will not start up reliably
The maximum crystal frequency that you can use with the PIC18F6520 depends on whether you have the PLL enabled. With it disabled you can use a crystal up to 25MHz (page 320 of the datasheet). With it enabled you are limited to the range 4 to 10MHz.

This certainly explains why the 32MHz crystal does not work reliably. I'm not sure why the 20MHz crytal doesn't work, but perhaps you have the PLL enabled? Or you are running the PIC from a much lower Vdd than 3.3V?