Can you pass resistance through analog mux?
The data sheet says that it has a resistance of 0.2 ohms with an analog signal of -5.5V to +5.5V. If the output of your variable potentiometer falls between that voltage range I see no reason why it wouldn't work. You are not really feeding "resistance" through it but current, no matter how small, but as long as your voltage and current are within the specs of the device it should work.

When you are "calibrating" the device you are sending a current through it and measuring the voltage drop to get at the resistance.