Is it good practice to include data cleaning or feature engineering steps in an sklearn pipeline to create a scalable pipeline?
One of the benefits of using scikit-learn's Pipeline is to appropriately apply the operations to both training and prediction datasets. The &quot;whole dataset&quot; you mention that is only the training data you currently have available. Scikit-learn's Pipeline can automatically handle additional data, such as production data.
Having two different pipelines can create unnecessary overhead and might lead to bugs in the system. It is best practices to have a single pipeline that handles all operations.