title: CountVectorize vocabulary specification for bigrams python
body: <p>I am trying to get a sparse matrix of term counts of a huge (~160.000) number of documents. </p>
  
  <p>I cleaned text and want to loop over all documents (i.e. count vectorize one at a time and append the resulting 1xN arrays. The following code works for the word by word case, but not for bigrams:</p>
  
  <pre><code>cv1 = sklearn.feature_extraction.text.CountVectorizer(stop_words=None,vocabulary=dictionary1)
  cv2 = sklearn.feature_extraction.text.CountVectorizer(stop_words=None,vocabulary=dictionary2)
  
  for row in range(start,end+1):
      report_name = fund_reports_table.loc[row, "report_names"]
      raw_report = open("F:/EDGAR_ShareholderReports/" + report_name, 'r', encoding="utf8").read()
  
      ## word for word
      temp = cv1.fit_transform([raw_report]).toarray()
      res1 = np.concatenate((res1,temp),axis=0)
  
      ## big grams
      bigram=set()
      sentences = raw_report.split(".")
      for line in sentences:
          token = nltk.word_tokenize(line)
          bigram = bigram.union(set(list(ngrams(token, 2)))  )
  
      temp = cv2.fit_transform(list(bigram)).toarray()
      res2=np.concatenate((res2,temp),axis=0)
  </code></pre>
  
  <p>Python returns </p>
  
  <pre><code>"AttributeError: 'tuple' object has no attribute 'lower'" 
  </code></pre>
  
  <p>presumably because the way I feed data into the bigram vectorizecounter is invalid. </p>
  
  <p>"raw_report" is a string. The one-by-word dictionary is: </p>
  
  <pre><code>dictionary1 =['word1', 'words2',...]
  </code></pre>
  
  <p>dictionary2 is similar but based on bigrams constructed by merging all bigrams of all documents (and keeping unique values, done in a previous) such that the resulting structure is </p>
  
  <pre><code>dictionary2 =[('word1','word2'),('wordn','wordm'),...]
  </code></pre>
  
  <p>The document bigram has the same structure, that's why I am puzzled why python won't accept the input. Is there a way to fix this or is my entire approach not very pythonic and starts to backfire?</p>
  
  <p>Thanks in advance for any help!</p>
  
  <p>Remark: I understand that I could do the entire process in a more elaborate CountVectorize command (i.e. the cleaning and tokenization and counting in one step), but I would much prefer too be able to do this myself (in order to see&amp;store intermediate outputs). Also I am afraid I run into memory issues given the large amount of text I use. </p>