# Feature importance with Text features

I would like to determine features importance in several models:

• support vector machine
• logistic regression
• Naive Bayes
• random forest

I read that I will need an agnostic model, so I have thought to use performance_importance (in python). My features look like

• Text (e.g., The pen is on the table, the sky is blue,...)
• Year (e.g., 2019, 2020,...)
• #_of_characters (e.g., 34, 67,...): this value comes from Text
• Party (e.g., National, Local, Green, ...)
• Over18 (e.g., 1, 0, ...) : this is a Boolean variable

My target variable is Voted.

In the pre-processing phase, I am using BoW and TF-IDF for Text, OneHotEncoder for Party, SimpleImputer for numerical. Using the following:

from sklearn.inspection import permutation_importance
import matplotlib.pyplot as plt

result = permutation_importance(clf, X_test, y_test, n_repeats=5, random_state=42, n_jobs=2)
sorted_idx = result.importances_mean.argsort()

plt.boxplot(result.importances[sorted_idx].T,
vert=False, labels=X.columns[sorted_idx]);


I am getting a similar output like the below(I forgot to include Over18, but it is just to give an idea of the output):

Although I have difficulties in interpreting the results, especially circles and negative values, I would like to understand if, in case of text classification, it makes sense to have Text for the importance and not, for example, the single words (unigrams, bigrams,...). For instance, in my example, I have ['The','pen','is','on','table','sky','blue']. Would it make more sense to understand how much each word contributes to the model instead of Text, or this is just considered in Text (where there are many words that contributes to the model), which is the most significant feature in the model?

UPDATE: for the different features I am using the following pre-processors:

categorical_preprocessing = OneHotEncoder(handle_unknown='ignore')

numeric_preprocessing = Pipeline([
('imputer', SimpleImputer(strategy='mean')
])

# CountVectorizer
text_preprocessing_cv =  Pipeline(steps=[
('CV',CountVectorizer())
])

# TF-IDF
text_preprocessing_tfidf = Pipeline(steps=[
('TF-IDF',TfidfVectorizer())
])


and then

preprocessing_cv = ColumnTransformer(
transformers=[
('text',text_preprocessing_cv, 'Text'),
('category', categorical_preprocessing, categorical_features),
('numeric', numeric_preprocessing, numerical_features)
], remainder='passthrough')

clf_nb = Pipeline(steps=[('preprocessor', preprocessing_cv),
('classifier', MultinomialNB())])

• Take a look at github.com/slundberg/shap It allows you to see explanation for a particular row of the data set and dependence plot for variable of interest – Yaroslaw Homenko May 13 at 6:15
• How did you fit the text feature i.e. The pen is on the table in the models – 10xAI May 14 at 8:03
• @10xAI, I used CountVectorizer (Bag of Words) and I am considering to use TF-IDF as well for a comparison. Please see the update question for this step – Math May 14 at 11:04
• @YaroslawHomenko unfortunately I will not be able to use shap. I have no GPU (only CPU) and I have had difficulties in installing it because of that – Math May 14 at 12:24
• @Math just to note, Shap does require GPU – Yaroslaw Homenko May 16 at 12:13

permutation_importance is considering the top-level features. It is permuting each one sequentially and learning the importance.
So, the inner encoding i.e. OHE/tfid is not visible to it.

To get the importance of components of the top-level feature, you should encode it separately and then pass the encoded data to the permutation_importance

• Get the pre-processed data using preprocessing_cv.fit_transform(X_train)
• Call your permutation_importance code on the above data and any model of your choice

## Edit

Adding the snippet. I am excluding ColumnTranformer as it is causing some issue.

data = {'Number':[1,2,3], 'Text':['pen is table', 'sky is blue','Sun is kool'], 'Cat':['A','B', 'C']}
df = pd.DataFrame(data)

categorical_preprocessing = OneHotEncoder(handle_unknown='ignore')
numeric_preprocessing = SimpleImputer(strategy='mean')
text_preprocessing_cv =  CountVectorizer()

text_tfid = text_preprocessing_cv.fit_transform(df['Text']).toarray()
num = numeric_preprocessing.fit_transform(df['Number'].values.reshape(-1, 1))
cat = categorical_preprocessing.fit_transform(df['Cat'].values.reshape(-1, 1)).toarray()

data = np.concatenate((cat,num,text_tfid), axis=1)
cols =  np.concatenate((categorical_preprocessing.get_feature_names(), text_preprocessing_cv.get_feature_names(), ['Num'])) # New cols name

df = pd.DataFrame(data, columns=cols) # Encoded DataFrame with col name


from sklearn.inspection import permutation_importance
import matplotlib.pyplot as plt
from sklearn.tree import DecisionTreeClassifier

clf = DecisionTreeClassifier().fit(df, [1,0,1])

result = permutation_importance(clf, df, [1,0,1], n_repeats=2, random_state=42)
sorted_idx = result.importances_mean.argsort()

plt.boxplot(result.importances[sorted_idx].T,
vert=False, labels=df.columns[sorted_idx]);


• Thank you so much, @10xAI. So in the permutation_importance, should I write permutation_importance(clf_nb, preprocessing_cv.fit_transform(X_train), y_train, n_repeats=10, random_state=42, n_jobs=2), on the train set and not on the test set, if I understood correctly? I was looking at the test set instead of train, but in this way I am replacing the above code I am getting this error message: TypeError: A sparse matrix was passed, but dense data is required. Use X.toarray() to convert to a dense numpy array. – Math May 16 at 15:37
• If I use todense() I get the following: ValueError: X has 1200 features, but ColumnTransformer is expecting 50 features as input. – Math May 16 at 15:44
• Not clf_nb as it is a pipeline. Create a separate model instane i.e. model=RandomForestClassifier() etc. – 10xAI May 16 at 16:00
• Thank you so much for your answer, 10xAI – Math May 18 at 13:31
{"extraction_info": {"found_math": true, "script_math_tex": 0, "script_math_asciimath": 0, "math_annotations": 0, "math_alttext": 0, "mathml": 0, "mathjax_tag": 0, "mathjax_inline_tex": 0, "mathjax_display_tex": 0, "mathjax_asciimath": 1, "img_math": 0, "codecogs_latex": 0, "wp_latex": 0, "mimetex.cgi": 0, "/images/math/codecogs": 0, "mathtex.cgi": 0, "katex": 0, "math-container": 0, "wp-katex-eq": 0, "align": 0, "equation": 0, "x-ck12": 0, "texerror": 0, "math_score": 0.45941251516342163, "perplexity": 4891.272219064631}, "config": {"markdown_headings": true, "markdown_code": true, "boilerplate_config": {"ratio_threshold": 0.18, "absolute_threshold": 10, "end_threshold": 15, "enable": true}, "remove_buttons": true, "remove_image_figures": true, "remove_link_clusters": true, "table_config": {"min_rows": 2, "min_cols": 3, "format": "plain"}, "remove_chinese": true, "remove_edit_buttons": true, "extract_latex": true}, "warc_path": "s3://commoncrawl/crawl-data/CC-MAIN-2021-31/segments/1627046154420.77/warc/CC-MAIN-20210803030201-20210803060201-00405.warc.gz"}