Unable to save the TF-IDF vectorizer I'm workig on multi-label classification problem. I'm facing issue while saving the TF-IDf verctorizer and as well as model using both pickle and joblib packages. 

Below is the code:

vectorizer = TfidfVectorizer(min_df=0.00009, max_features=200000, smooth_idf=True, norm="l2", \
                         tokenizer = lambda x: x.split(), sublinear_tf=False, ngram_range=(1,3))
x_train_multilabel = vectorizer.fit_transform(x_train)
x_test_multilabel = vectorizer.transform(x_test)

classifier = OneVsRestClassifier(SGDClassifier(loss='log', alpha=0.00001, penalty='l1'), n_jobs=-1)
classifier.fit(x_train_multilabel, y_train)
predictions = classifier.predict(x_test_multilabel)


Error Message while saving the TF-IDF vectozier. 




Any suggestions ?
Thanks in advance.
The issue is due to your lamda function with the tokenizer key word argument.

&gt;&gt;&gt; from sklearn.feature_extraction.text import TfidfVectorizer
&gt;&gt;&gt; from joblib import dump
&gt;&gt;&gt; t = TfidfVectorizer()
&gt;&gt;&gt; dump(t, 'tfidf.pkl')
['tfidf.pkl']


No issues. Now let's pass a lambda function to tokenizer

&gt;&gt;&gt; t = TfidfVectorizer(tokenizer=lambda x: x.split())
&gt;&gt;&gt; dump(t, 'tfidf.pkl')


Which throws the following error:


  _pickle.PicklingError: Can't pickle  at 0x100e18b90>: it's not found as main.


To get around this, create a function to split the text:

&gt;&gt;&gt; def text_splitter(text):
...     return text.split()


Try dumping again:

&gt;&gt;&gt; t = TfidfVectorizer(tokenizer=text_splitter)
&gt;&gt;&gt; dump(t, 'tfidf.pkl')
['tfidf.pkl']


Now you can you save the vectorizer.