Predicting unseen data on Target variable based Encoding Technique
For predictions on unseen data, you should simply omit the target encoding from your pipeline. You can therefore implement two versions of the pipeline.
This would be your training/testing/cross-validation pipeline:
User uploads training data -- Data Cleaning -- Encoding(Target Encoder) -- Training the model -- result
Note: use fit_transform on the encoding when running training data, and transform when running test or validation data to avoid data leakage
And this would be your prediction pipeline:
User uploads testing data -- Data Cleaning -- predictions using trained models
I am working on an automated ML (Regression) algorithm where the flow of process is:
User uploads a data -- Data Cleaning --  Encoding(Target Encoder) -- Fitting model -- results.
I am completely fine until this point, my confusion is when the user wants to test this in an unseen data without target variable, then I need to again perform Data cleaning -- Encoding and encoding technique I have used while fitting the model can work only if there is a target variable (unseen data will not have a target variable) and I cannot change the Encoding technique on unseen data as the testing data needs to go through the same procedure as the data used while fitting the model as per my knowledge.
Could someone please help me in finding a way to overcome this issue or any suggestions would be of great help.
Thanks in advace.