Controlling PWM with an external oscillator and a JFET
The goal of your circuit is to change the voltage on the comparator's inverting node.
Consider using a summing amplifier to add the potentiometer offset to the CV signal instead of using a JFET to attenuate the potentiometer's wiper voltage.
This is how I do variable pulse width in my synthesizer:

The triangle input is 5Vpp. Nominal voltage to the comparator's noninverting input is 0V, giving 50% pulse width. It is variable from ~0% to ~100% (-2.5V to +2.5V) by both the potentiometer or external CV.