Accommodating Gain Elements in PLL Loop
A 'loop filter' does not 'keep the loop stable'. At best it allows the loop to be stable. However, if you get it wrong, it will destablise the loop.
I notice your loop filter appears to claim a bandwidth of 475 Hz. That is far too close to your hoped-for 400 Hz, and will likely cause instability.
Paradoxically, the loop filter does not set the loop bandwidth. That's done by the loop gain.
First, remove all your filters, or at least remove their effect. Remove the capacitors from any high frequency low pass filters. Replace any capacitors in the low frequency integrator by a short if DC conditions allow. Otherwise replace by a very large (infinite) capacitor. This may extend your lock time, but that will be fixed later when we re-introduce the filters
Once your PLL is clean of filters, we can calculate the gain. Break the loop at a convenient point, anywhere will do.
Worked Example
Break the loop at the input to your 4002 board. Let's apply 1 radian deviation. This will be divided by the 'coarse tuning steps' divider, whatever N that is. Phase gain is 1/N radians/rad.
Now there's a current-output PSD. Take care with the gain of this one, the output current might be specified per radian, or per cycle, or per other phase step. Use an appropriate factor to get k into Amps/radian. Current gain is now k/N amps/rad.
Now into the loop filter. C1 shunts the PSD current. This lowpass filter is above the loop bandwidth, so we open circuit C1, and completely ignore it, for the moment.
The rest of the loop filter is both trickier and simpler than it looks. The opamp end of R1 is held at ground by the action of the opamp virtual ground input. The voltage gain so far is therefore k.R1/N V/rad.
Now deal with C2. This is a lift in gain below the loop bandwidth, so we replace it with a short circuit, or an infinite value capacitor, either way it doesn't affect the gain.
The gain of the opamp stage is now R2/R1, so the voltage gain to the output is (R2/R1).(k.R1/N) = k.R2/N V/rad.
However, we could have got there quicker noting that all the DC current that flows through R1 also flows through R2, so the transresistance gain is R2 anyway. The voltage gain is now R2.k/N V/rad. (1)
This voltage drives the VFO. The VFO has a sensitivity of p Hz/V, so the frequency gain is p.R2.k/N Hz/rad.
The mixer stage might be inverting, however it's not going to change the magnitude of the gain.
Finally, we need to convert the frequency from the mixer to phase at the 4002 input, in the implicit integrator that is the basis of every PLL. This is easy to get wrong, I did first time round.
A frequency deviation of 1 Hz, at a modulation rate of 1 Hz, gives rise to a phase deviation of 1 radian. The gain of the integrator is therefore 1 rad/Hz at 1 Hz, or 1 mrad/Hz at 1 kHz. (2)
Perhaps the easiest way to handle this is to compute the gain round the loop at 1 Hz, and then the loop gain will numerically equal the final loop bandwidth.
So to complete the calculation, at 1 Hz, the integrator gain is 1 rad/Hz, so the entire gain round the loop is p.R2.k/N. This will drop to unity at a frequency of p.R2.k/N Hz, which is the criterion for the loop bandwidth.
Now Handle the Loop Filters
This loop will lock, and will be super-stable. It will have limited rejection of reference high frequency noise, so now we bring the high pass filter C1.R1 down in frequency to improve it. If we need even more rejection than a single RC gives us, then we can use a higher order lowpass filter. Fractional-N synthesisers often use a much higher order passive filter after the PSD to deal with the rising spectrum of hash that's produced by the process.
It will have poor tracking of the reference, so now we bring the broken integrator C2.R2 up in frequency.
The two filters affect the loop gain in opposite directions, so the effect on loop gain and so loop bandwidth is small.
As a crude but useful rule of thumb, a single pole filter will have 45° phase shift at its corner frequency, 30° at twice that, 15° at four times that and so on. You therefore want your high frequency filter to have a corner of around 4*400 = 1600 Hz. You want your low frequency broken integrator to have a break frequency of around 400/4 = 100 Hz.
If you did have a higher order lowpass filter, then you would still need to budget for a small phase shift down at 400 Hz. You could move your low frequency filter down to allow for more phase shift from your high frequency filter. It's the total phase shift budget at the loop bandwidth that matters.
One on the benefits of having both C1 and C2 filters is that the total phase shift over the range 200 Hz to 800 Hz doesn't change by much. This is important if the VFO you are using has a non-linear Hz/V tuning curve, or you adjust the coarse steps divider. As these appear in the loop bandwidth expression, the loop bandwidth will change as these change. Having little change in excess phase shift over a range of loop bandwidths means that stability and settling time will remain largely unaffected by the changes in loop bandwidth.
(1) Does that mean that R1 doesn't do anything? It doesn't do anything at the loop bandwidth, but it does set the time constant of the high frequency filter with C1. It's odd interactions like this that make hope'n'poke adjustment of loop filter components so frustrating when trying to adjust a PLL into stability. Generally, you have to sit down and work it through as in the example above to be able to set the unity gain frequency, the high filter break, and the low filter break independently and correctly.
(2) A simple way to remember this from sort of first principles is to note that if there's a frequency deviation of 1 Hz that's held for 1 second, then there will be a phase change of 1 cycle, or 2pi radians. Taking a peak 1 Hz frequency deviation, and modulating at 1 Hz will clearly give much less phase deviation. It feels to me that 1/6th cycle, one radian, is about the right amount of 'much less', which is confirmed if we do the sums properly.