Feature importance with Text features I would like to determine features importance in several models:

support vector machine
logistic regression
Naive Bayes
random forest

I read that I will need an agnostic model, so I have thought to use performance_importance (in python).
My features look like

Text (e.g., The pen is on the table, the sky is blue,...)
Year (e.g., 2019, 2020,...)
#_of_characters (e.g., 34, 67,...): this value comes from Text
Party (e.g., National, Local, Green, ...)
Over18 (e.g., 1, 0, ...) : this is a Boolean variable

My target variable is Voted.
In the pre-processing phase, I am using BoW and TF-IDF for Text, OneHotEncoder for Party, SimpleImputer for numerical.
Using the following:
from sklearn.inspection import permutation_importance
import matplotlib.pyplot as plt

result = permutation_importance(clf, X_test, y_test, n_repeats=5, random_state=42, n_jobs=2)
sorted_idx = result.importances_mean.argsort()

plt.boxplot(result.importances[sorted_idx].T,
            vert=False, labels=X.columns[sorted_idx]);

I am getting a similar output like the below(I forgot to include Over18, but it is just to give an idea of the output):

Although I have difficulties in interpreting the results, especially circles and negative values, I would like to understand if, in case of text classification, it makes sense to have Text for the importance and not, for example, the single words (unigrams, bigrams,...).
For instance, in my example, I have ['The','pen','is','on','table','sky','blue']. Would it make more sense to understand how much each word contributes to the model instead of Text, or this is just considered in Text (where there are many words that contributes to the model), which is the most significant feature in the model?
UPDATE: for the different features I am using the following pre-processors:
categorical_preprocessing = OneHotEncoder(handle_unknown='ignore')

numeric_preprocessing = Pipeline([
    ('imputer', SimpleImputer(strategy='mean')
])

# CountVectorizer
text_preprocessing_cv =  Pipeline(steps=[
    ('CV',CountVectorizer())
]) 

# TF-IDF
text_preprocessing_tfidf = Pipeline(steps=[
    ('TF-IDF',TfidfVectorizer())       
]) 

and then
preprocessing_cv = ColumnTransformer(
    transformers=[
        ('text',text_preprocessing_cv, 'Text'),
        ('category', categorical_preprocessing, categorical_features),
        ('numeric', numeric_preprocessing, numerical_features)
], remainder='passthrough')

clf_nb = Pipeline(steps=[('preprocessor', preprocessing_cv),
                      ('classifier', MultinomialNB())])
permutation_importance is considering the top-level features. It is permuting each one sequentially and learning the importance.
So, the inner encoding i.e. OHE/tfid is not visible to it.
To get the importance of components of the top-level feature, you should encode it separately and then pass the encoded data to the permutation_importance

Get the pre-processed data using preprocessing_cv.fit_transform(X_train)
Call your permutation_importance code on the above data and any model of your choice

Edit
Adding the snippet. I am excluding ColumnTranformer as it is causing some issue.
data = {'Number':[1,2,3], 'Text':['pen is table', 'sky is blue','Sun is kool'], 'Cat':['A','B', 'C']}
df = pd.DataFrame(data)

categorical_preprocessing = OneHotEncoder(handle_unknown='ignore')
numeric_preprocessing = SimpleImputer(strategy='mean')
text_preprocessing_cv =  CountVectorizer()

text_tfid = text_preprocessing_cv.fit_transform(df['Text']).toarray()
num = numeric_preprocessing.fit_transform(df['Number'].values.reshape(-1, 1))
cat = categorical_preprocessing.fit_transform(df['Cat'].values.reshape(-1, 1)).toarray()

data = np.concatenate((cat,num,text_tfid), axis=1)
cols =  np.concatenate((categorical_preprocessing.get_feature_names(), text_preprocessing_cv.get_feature_names(), ['Num'])) # New cols name

df = pd.DataFrame(data, columns=cols) # Encoded DataFrame with col name


from sklearn.inspection import permutation_importance
import matplotlib.pyplot as plt
from sklearn.tree import DecisionTreeClassifier

clf = DecisionTreeClassifier().fit(df, [1,0,1])

result = permutation_importance(clf, df, [1,0,1], n_repeats=2, random_state=42)
sorted_idx = result.importances_mean.argsort()

plt.boxplot(result.importances[sorted_idx].T,
            vert=False, labels=df.columns[sorted_idx]);