title: signed integer is greater than maximum in scikit-learn in python
body: <p>I am working on sentiment analysis of around 30,000 tweets. python version is 2.7 on linux. In the training phase I am using <code>nltk</code> as a wrapper for <code>sklearn</code> library to apply different Classifiers such as Naive Bayes, LinearSVC, Logistic regression , etc. </p>
  
  <p>It works fine when the number of tweets are like 10,000 but now I received error for 30,000 tweets on classifying Bigrams with Multinomial naive bayes in <code>sklearn</code>. Here is part of the implementation code after pre-processing and dividing to train and test sets : </p>
  
  <pre><code>import nltk
  from nltk.classify.scikitlearn import SklearnClassifier
  from sklearn.naive_bayes import MultinomialNB,
  
  training_set = nltk.classify.util.apply_features(extractFeatures, trainTweets)
  testing_set = nltk.classify.util.apply_features(extractFeatures, testTweets)
  
  MNB_classifier = SklearnClassifier(MultinomialNB())
  MNB_classifier.train(training_set)
  MNBAccuracy = nltk.classify.accuracy(MNB_classifier, testing_set)*100
  print "-------- MultinomialNB --------"
  print "RESULT : Matches  " + str(int((testSize*MNBAccuracy)/100)) + ":"+ str(testSize)
  print "MNB accuracy percentage:" + str(MNBAccuracy)
  print ""
  </code></pre>
  
  <p>here the Error: </p>
  
  <pre><code>Traceback (most recent call last):
    File "/home/sb402747/Desktop/Sentiment/sentiment140API/analysing/Classifier.py", line 83, in &lt;module&gt;
      MNB_classifier.train(training_set)
    File "/home/sb402747/.local/lib/python2.7/site-packages/nltk/classify/scikitlearn.py", line 115, in train
      X = self._vectorizer.fit_transform(X)
    File "/home/sb402747/.local/lib/python2.7/site-packages/sklearn/feature_extraction/dict_vectorizer.py", line 226, in fit_transform
      return self._transform(X, fitting=True)
    File "/home/sb402747/.local/lib/python2.7/site-packages/sklearn/feature_extraction/dict_vectorizer.py", line 176, in _transform
      indptr.append(len(indices))
  OverflowError: signed integer is greater than maximum
  </code></pre>
  
  <p>I guess the reason is because the number of indices in array is more that the maximum allowed for it on <code>dict_vectore.py</code>. I even tried to change the type of indices in <code>dict_vectorizer.py</code> from <code>i</code> to <code>l</code> but it didn't solve my problem and received this error: </p>
  
  <pre><code>Traceback (most recent call last):
    File "/home/sb402747/Desktop/Sentiment/ServerBackup26-02-2016/analysing/Classifier.py", line 84, in &lt;module&gt;
      MNB_classifier.train(training_set)
    File "/home/sb402747/.local/lib/python2.7/site-packages/nltk/classify/scikitlearn.py", line 115, in train
      X = self._vectorizer.fit_transform(X)
    File "/home/sb402747/.local/lib/python2.7/site-packages/sklearn/feature_extraction/dict_vectorizer.py", line 226, in fit_transform
      return self._transform(X, fitting=True)
    File "/home/sb402747/.local/lib/python2.7/site-packages/sklearn/feature_extraction/dict_vectorizer.py", line 186, in _transform
      shape=shape, dtype=dtype)
    File "/rwthfs/rz/SW/UTIL.common/Python/2.7.9/x86_64/lib/python2.7/site-packages/scipy/sparse/compressed.py", line 88, in __init__
      self.check_format(full_check=False)
    File "/rwthfs/rz/SW/UTIL.common/Python/2.7.9/x86_64/lib/python2.7/site-packages/scipy/sparse/compressed.py", line 167, in check_format
      raise ValueError("indices and data should have the same size")
  ValueError: indices and data should have the same size
  </code></pre>
  
  <p>then discarded it and changed it back to <code>i</code> again. How can I solve this problem? </p>