PCIe Gen2 PLL lock issue
Place a Spectrum Analyzer on PLL output, and examine the various frequency components causing the jitter. Then identify the sources. Then decide how to reduce the sources of the trash.

1) does your 100MHz Reference clock remain CLEAN? if interferers are linearly added to the 100MHz signal, then converted into a square-wave inside the IC, you will have increased edge jitter. Again, use a spectrum analyzer and identify the frequency components. You will need to account for up-multiplication and for down-division, in diagnosing the components.

2) is your VDD clean? some circuits (logic gates in particular) have ZERO (OK maybe 6dB) Power Supply Rejection. Thus 10 milliVolts of VDD trash, from the switching power supply, causes the zero-crossing point of logic gates to wander in time; thus  jitter can be ADDED after the PLL, during logic-flipflop division ratios.

3) do you use Ground planes, to protect the signals? As the 100MHz Fref moves about the PCB, are there any slits under the trace? Does the trace run near a switching power supply?