Why does the voltage across the diode drop with parallel resistor value in this circuit?
As long as the diode is not conducting, the voltage VR1 is determined by the voltage divider R1 over R2. Increasing R1 will increase VR1, as soon as VR1 reached the diode forward voltage( approx 0.7V) the diode start conducting and will clip the voltage to Vf.

mathematically:
value of R1 @ which the voltage will be clipped:

R1/(R1+R2) * Vcc = Vf So
R1 = (Vf*R2)/(Vcc - Vf) = (approx) 620 ohm

Of course this is the practical diode model, which is a good approximation. The complete diode model will be more accurate but will also introduce some more math