title: Pickle Tfidfvectorizer along with a custom tokenizer
body: <p>I'm using a costume tokenizer to pass to TfidfVectorizer. That tokenizer depends on an external class TermExtractor, which is in another file.</p>
  
  <p>I basically want to build a TfidVectorizer based on certain terms, and not all single words/tokens.</p>
  
  <p>Here is to code to it:</p>
  
  <pre><code>from sklearn.feature_extraction.text import TfidfVectorizer
  from TermExtractor import TermExtractor
  
  extractor = TermExtractor()
  
  def tokenize_terms(text):
      terms = extractor.extract(text)
      tokens = []
      for t in terms:
          tokens.append('_'.join(t))
      return tokens
  
  
  def main(): 
      vectorizer = TfidfVectorizer(lowercase=True, min_df=2, norm='l2', smooth_idf=True, stop_words=stop_words, tokenizer=tokenize_terms)
      vectorizer.fit(corpus)
      pickle.dump(vectorizer, open("models/terms_vectorizer", "wb"))
  </code></pre>
  
  <p>This runs fine, but whenever I want to re-use this TfidfVectorizer and load it with pickle, I get an error:</p>
  
  <pre><code>vectorizer = pickle.load(open("models/terms_vectorizer", "rb"))
  
  Traceback (most recent call last):
    File "./train-nps-comments-classifier.py", line 427, in &lt;module&gt;
      main()
    File "./train-nps-comments-classifier.py", line 325, in main
      vectorizer = pickle.load(open("models/terms_vectorizer", "rb"))
    File "/usr/lib/python2.7/pickle.py", line 1378, in load
      return Unpickler(file).load()
    File "/usr/lib/python2.7/pickle.py", line 858, in load
      dispatch[key](self)
    File "/usr/lib/python2.7/pickle.py", line 1090, in load_global
      klass = self.find_class(module, name)
    File "/usr/lib/python2.7/pickle.py", line 1126, in find_class
      klass = getattr(mod, name)
  AttributeError: 'module' object has no attribute 'tokenize_terms'
  </code></pre>
  
  <p>How does Python pickle works when there are dependent classes?</p>