How to calculate Variable Importance in SVM regression models
Question: How do I calculate the variable importance of an
[SVM](https://en.wikipedia.org/wiki/Support_vector_machine) regression model
implemented in Python?

At least, if an already-implemented function does not exist, I would like some
hints how to calculate it theoretically. I have already searched in the
literature without any success. Any idea?


Answer: You can use "l1" as the [penalty function](http://scikit-
learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html) to get a sparse
model. See [here](http://stats.stackexchange.com/questions/2179/variable-
importance-from-svm) for details. Then just inspect the coefficients of the
learned model.

Below we use the [iris
dataset](https://en.wikipedia.org/wiki/Iris_flower_data_set) as an example.

    
    
    from sklearn import svm, datasets
    iris = datasets.load_iris()
    X = iris.data
    y = iris.target
    linear_svc = svm.LinearSVC(penalty='l1',dual=False).fit(X,y)
    >>> linear_svc.coef_.shape
    (3, 4) 
    # Note the three rows corresponds to the three classes in iris dataset
    # the four columns corresponds to the four features.
    >>> linear_svc.coef_
    ## feature  sepal-l     sepal-w      petal-l      petal-w
    array([[ 0.09528818,  0.62495604, -0.90205483,  0.        ],   # 'setosa',  
           [ 0.        , -0.93523507,  0.40433182, -0.89729973],   # 'versicolor',
           [-0.65391116, -0.94345497,  1.29683221,  2.10726403]])  # 'virginica'
    

When inspecting the variable importance, you should refer to
[this](https://upload.wikimedia.org/wikipedia/commons/5/56/Iris_dataset_scatterplot.svg)
graph at wikipedia.

For example, the first class (first row in `linear_svc.coef_` \- "setosa" (red
in below graph: x-axis is sepal-width, y-axis is petal-length) can be
perfectly separated (with a linear separator) from the other two classes with
only the two features in the middle. I think that's why the fourth feature has
a zero coefficient and the first feature's coefficient is very small. Because
given the two features - sepal-width and petal-length, it's sufficient to tell
whether the species is "setosa" or not.

[![enter image description
here](http://i.stack.imgur.com/xVazf.png)](http://i.stack.imgur.com/xVazf.png)

On the other hand, "versicolor" and "virginica" are harder to separate from
each other using a linear separator. Hence 'versicolor' has three non-zero
coefficient and 'virginica' has four.

**EDIT** : I just noticed you are talking about an SVM _regression_ model. It
should be pretty similar. But you might want to use [`SVR`](http://scikit-
learn.org/stable/modules/generated/sklearn.svm.LinearSVR.html).

