Machine Learning Pipeline Automation in Python
Machine Learning Pipeline Automation in Python

Model Selection, Preprocessing and Hyperparameters Fine-tuning

Source: unsplash.com

Most Data Scientists and Machine Learning Engineers will agree that having an efficient workflow for evaluating and choosing the best model is of extreme importance.

However, as I explained in a previous article, there are often several algorithms to choose from to solve our problem and many of them (e.g., support vector classifier and random forests) need hyperparameters finetuning order to select the best performing model. On top of this, very often we will need to apply some preprocessing to our data.

Considering the vast amount of available algorithms, the main skill of the modern data scientist consists of a focused approach to model selection, with a defined pipeline that she will apply as a first step to decide what model is the best approach for a particular problem.

In this article I am going to discuss effective workflows for model selection, using cross-validation to evaluate performance and finetuning hyperparameters on the fly. I will use Python’s popular library Scikit-Learn for my coding examples.

If you are a beginner and need an in-depth explanation of cross-validation please have a look at my previous article before you continue reading, as here we are going to assume that you are familiar with overfitting and cross-validation.

The first concept I would like to introduce here is “exhaustive search”.

Exhaustive Search:

Exhaustive Search methods like Grid Search and Randomized Search will try a range of values provided by users for specific hyperparameters and come up with the best values.

GridSearchCV is a brute-force method provided by the popular Python library Scikit-Learn. It approaches model selection using cross-validation. It needs the user to provide specific parameter values, then it trains a model using every value and/or combination of values. The model with the best performance score is selected as the best model.

The problem with grids is that they are computationally expensive, a more efficient method than GridSearchCV’s brute-force search is RandomizedSearchCV.

RandomizedSearchCV is similar to grids but implements random samples of hyperparameters taken from a given distribution.

With RandomizedSearchCV, if we specify a distribution (normal, uniform …) Scikit-Learn will randomly sample without replacement hyperparameter values from that distribution. That’s faster than grid search.

Let’s see an example with the popular Iris dataset:

Now imagine we want to try several classification models with their correspondent hyperparameters and pick the best for our problem. How can we incorporate this into our pipeline?

Well, the cool thing with Scikit-Learn is that this can be automated with a simple solution: building a dictionary of models and hyperparameters.

We can build a search space with our desired models and pass it to either GridSearchCV or RandomizedSearchCV methods

See the code below for an example of this comparing Random Forests and Logistic Regression. The dictionary is called Search Space.

Preprocessing

Finally, we can also add a preprocessing step to our workflow on top of cross-validation, model selection, and hyperparameters fine-tuning.

Very often we will need to preprocess our data before using it to train a model. We have to be careful to properly handle preprocessing when conducting model selection.

First, GridSearchCV uses cross-validation to determine which model has the highest performance. However, in cross-validation we are in effect pretending that the fold held out, as the test set is not seen, and thus not part of fitting any preprocessing steps (e.g., scaling or standardization).

For this reason, we cannot preprocess the data and then run GridSearchCV. Rather, the preprocessing steps must be a part of the set of actions taken by GridSearchCV. While this might appear complex, the reality is that Scikit-Learn makes it simple.

The method called FeatureUnion combines multiple preprocessing actions. We are going to show an example of FeatureUnion combining two typical preprocessing steps: standardize the feature values (StandardScaler) and Principal Component Analysis (PCA).

We then include preprocessing into a pipeline with our learn‐ ing algorithm. The end result is that this allows us to outsource the proper fitting, transforming, and training the models with combinations of hyperparameters to Scikit-Learn.

Just like this:

Some preprocessing methods have their own parameters, which often have to be supplied by the user. For example, dimensionality reduction using PCA requires the user to define the number of principal components to use, which is defined in the search space like this: features__pca__n_components’: [1, 2, 3]. We pass it to the search space to indicate that we want to discover if one, two, or three principal components produced the best model.

finally, we can view the preprocessing values that produced the best model. For example, we can see the best number of principal components calling the method .get_params() like in the last line of our example.

Alright so that was all for today, hope you found this workflow useful to incorporate it as the first step in your data analysis.

Happy coding!
['Diego Salinas']
['Programming', 'Machine Learning', 'Artificial Intelligence', 'Python', 'Data Science']