How does a switch's series voltage affect discharge time in a relaxation circuit?
Due to the fact that in your circuit \$V_∞ = 20V\$ instead of \$0V\$.
You need to use this general formula for the capacitor charging/discharging phase:
$$V_C(t) = V∞ + (V_{start} - V∞) \times \left(e^{\frac{-t}{RC}}\right)$$
Where:
\$V_{start}\$  initial capacitor voltage.
\$V∞\$   steady-state final voltage.
Because now you have been using the simplified version that assumes \$V_∞ = 0V\$.
And if we solve it for the time we get this:
$$T = RC \times \ln \frac{V_{start} - V∞}{V_C - V∞} = 1\text{ms} \times \ln\frac{100V -20V}{50V - 20V} \approx 1\text{ms} \times 0.98 \approx 0.98\text{ms}$$