Do I need to encode samples during inference?
Yes the encoders need to be rerun during inference time. You basically need to run the exact same data preprocessing steps prior to training on the unseen observations. Without them your model will not be to recognize unseen data and hence fair poorly.