Multicollinearity(Variance Inflation Factor). Variables to remove before doing a model I am doing an exercise of a Machine Learning System module in python that takes a dataset of cars (cylinders, year, consumption....) and asks for a model, being the variable to predict the consumption of gasoline. As it has three categorical variables, I have generated the dummies.



In the exercise I need to eliminate the variables with multicollinearity, so I used the method showed on my course notes:

from sklearn.linear_model import LinearRegression

def calculateVIF(data):
    features = list(data.columns)
    num_features = len(features)

    model = LinearRegression()

    result = pd.DataFrame(index = ['VIF'], columns = features)
    result = result.fillna(0)

    for ite in range(num_features):
        x_features = features[:]
        y_featue = features[ite]
        x_features.remove(y_featue)

        x = data[x_features]
        y = data[y_featue]

        model.fit(data[x_features], data[y_featue])

        result[y_featue] = 1/(1 - model.score(data[x_features], data[y_featue]))

    return result


Then if I launch the method it calculates a coefficient for each variable:



In my course notes it is said:


$VIF&gt;5$ is a high value.
$VIF&gt;10$ is a very high value


What should I do? I need to remove the variables that have a $VIF&gt;10$ before executing the model?

The problem I see, for my categorical variable cylinders, is only cylinders_5 has a VIF under 10 so should I remove the others and leave cyclinders_5?
Here is a code I have written to handle Multicollinearity in a dataset. This code snippet is able to handle the following listed items:

Multicollinearity using Variable Inflation Factor (VIF), set to a default threshold of 5.0
You just need to pass the dataframe, containing just those columns on which you want to test multicollinearity.
This function will drop those columns which contains just 1 value. For a bit more further details on this point, please have a look my answer on How to run a multicollinearity test on a pandas dataframe?.
The calculation of VIF is parallelized over multiple cores.

    from joblib import Parallel, delayed
    from statsmodels.stats.outliers_influence import variance_inflation_factor
    
    def removeMultiColl(data, vif_threshold = 5.0):
        for i in data.columns:
            if data[i].nunique() == 1:
                print(f&quot;Dropping {i} due to just 1 unique value&quot;)
                data.drop(columns = i, inplace = True)
        drop = True
        col_list = list(data.columns)
        while drop == True:
            drop = False
            vif_list = Parallel(n_jobs = -1, verbose = 5)(delayed(variance_inflation_factor)(data[col_list].values, i) for i in range(data[col_list].shape[1]))
            max_index = vif_list.index(max(vif_list))
            if vif_list[max_index] &gt; vif_threshold:
                print(f&quot;Dropping column : {col_list[max_index]} at index - {max_index}&quot;)
                del col_list[max_index]
                drop = True
        print(&quot;Remaining columns :\n&quot;, list(data[col_list].columns))
        return data[col_list]

Good luck !