Difference between delayed down-converted signal and a down-converted delayed signal
Let your original signal be (leaving out any constant multiplicative factor):

$$s(t)=e^{i\omega_ct}m(t)$$

Then the down-converted signal is

$$\tilde{s}(t)=e^{i(\omega_c-\omega_{lo})t}m(t)$$

and its delayed version is

$$\tilde{s}(t-\tau)=e^{i(\omega_c-\omega_{lo})(t-\tau)}m(t-\tau)$$

whereas the delayed and down-converted signal is

$$s(t-\tau)e^{-i\omega_{lo}t}=e^{i\omega_c(t-\tau)}e^{-i\omega_{lo}t}m(t-\tau)=
\tilde{s}(t-\tau)e^{-i\omega_{lo}\tau}$$

So the correction factor $e^{-i\omega_{lo}\tau}$ indeed depends on both $\omega_{lo}$ and the delay $\tau$.