<p dir="auto"><code class="notranslate">sklearn.feature_extraction.text.HashingVectorizer</code> uses <code class="notranslate">sklearn/feature_extraction/_hashing.pyx</code> which has a signed 32bit variable as a counter which overflows on my dataset and causes the following error:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Traceback (most recent call last):                                                                                                                           [0/1999]
  File &quot;/home/erg/foo/foo.py&quot;, line 109, in fit_transform
    return self.foo.fit_transform(X)
  File &quot;/usr/lib/python3.6/site-packages/sklearn/pipeline.py&quot;, line 290, in fit_transform
    Xt, fit_params = self._fit(X, y, **fit_params)
  File &quot;/usr/lib/python3.6/site-packages/sklearn/pipeline.py&quot;, line 222, in _fit
    **fit_params_steps[name])
  File &quot;/usr/lib/python3.6/site-packages/sklearn/externals/joblib/memory.py&quot;, line 362, in __call__
    return self.func(*args, **kwargs)
  File &quot;/usr/lib/python3.6/site-packages/sklearn/pipeline.py&quot;, line 589, in _fit_transform_one
    res = transformer.fit_transform(X, y, **fit_params)
  File &quot;/usr/lib/python3.6/site-packages/sklearn/base.py&quot;, line 518, in fit_transform
    return self.fit(X, **fit_params).transform(X)
  File &quot;/usr/lib/python3.6/site-packages/sklearn/feature_extraction/text.py&quot;, line 519, in transform
    X = self._get_hasher().transform(analyzer(doc) for doc in X)
  File &quot;/usr/lib/python3.6/site-packages/sklearn/feature_extraction/hashing.py&quot;, line 167, in transform
    shape=(n_samples, self.n_features))
  File &quot;/usr/lib/python3.6/site-packages/scipy/sparse/compressed.py&quot;, line 98, in __init__
    self.check_format(full_check=False)
  File &quot;/usr/lib/python3.6/site-packages/scipy/sparse/compressed.py&quot;, line 167, in check_format
    raise ValueError(&quot;indices and data should have the same size&quot;)
ValueError: indices and data should have the same size"><pre class="notranslate"><code class="notranslate">Traceback (most recent call last):                                                                                                                           [0/1999]
  File "/home/erg/foo/foo.py", line 109, in fit_transform
    return self.foo.fit_transform(X)
  File "/usr/lib/python3.6/site-packages/sklearn/pipeline.py", line 290, in fit_transform
    Xt, fit_params = self._fit(X, y, **fit_params)
  File "/usr/lib/python3.6/site-packages/sklearn/pipeline.py", line 222, in _fit
    **fit_params_steps[name])
  File "/usr/lib/python3.6/site-packages/sklearn/externals/joblib/memory.py", line 362, in __call__
    return self.func(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/sklearn/pipeline.py", line 589, in _fit_transform_one
    res = transformer.fit_transform(X, y, **fit_params)
  File "/usr/lib/python3.6/site-packages/sklearn/base.py", line 518, in fit_transform
    return self.fit(X, **fit_params).transform(X)
  File "/usr/lib/python3.6/site-packages/sklearn/feature_extraction/text.py", line 519, in transform
    X = self._get_hasher().transform(analyzer(doc) for doc in X)
  File "/usr/lib/python3.6/site-packages/sklearn/feature_extraction/hashing.py", line 167, in transform
    shape=(n_samples, self.n_features))
  File "/usr/lib/python3.6/site-packages/scipy/sparse/compressed.py", line 98, in __init__
    self.check_format(full_check=False)
  File "/usr/lib/python3.6/site-packages/scipy/sparse/compressed.py", line 167, in check_format
    raise ValueError("indices and data should have the same size")
ValueError: indices and data should have the same size
</code></pre></div>
<p dir="auto">Applying this patch and running my code gives an error:</p>
<div class="highlight highlight-source-diff notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="diff --git a/sklearn/feature_extraction/_hashing.pyx b/sklearn/feature_extraction/_hashing.pyx
index e39aeafa0..b80d932cb 100644
--- a/sklearn/feature_extraction/_hashing.pyx
+++ b/sklearn/feature_extraction/_hashing.pyx
@@ -79,4 +79,6 @@ def transform(raw_X, Py_ssize_t n_features, dtype, bint alternate_sign=1):
         indptr[len(indptr) - 1] = size
 
     indices_a = np.frombuffer(indices, dtype=np.int32)
+    if not len(indices_a) == size:
+        raise ValueError(&quot;len indices_a: &quot; + str(len(indices_a)) + &quot;, size: &quot; + str(size))"><pre class="notranslate"><span class="pl-c1">diff --git a/sklearn/feature_extraction/_hashing.pyx b/sklearn/feature_extraction/_hashing.pyx</span>
index e39aeafa0..b80d932cb 100644
<span class="pl-md">--- a/sklearn/feature_extraction/_hashing.pyx</span>
<span class="pl-mi1">+++ b/sklearn/feature_extraction/_hashing.pyx</span>
<span class="pl-mdr">@@ -79,4 +79,6 @@</span> def transform(raw_X, Py_ssize_t n_features, dtype, bint alternate_sign=1):
         indptr[len(indptr) - 1] = size
 
     indices_a = np.frombuffer(indices, dtype=np.int32)
<span class="pl-mi1"><span class="pl-mi1">+</span>    if not len(indices_a) == size:</span>
<span class="pl-mi1"><span class="pl-mi1">+</span>        raise ValueError("len indices_a: " + str(len(indices_a)) + ", size: " + str(size))</span></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Traceback (most recent call last):                                                                                                                                                            
  File &quot;/usr/lib/python3.6/site-packages/sklearn/pipeline.py&quot;, line 289, in fit_transform                                                                                                     
    Xt, fit_params = self._fit(X, y, **fit_params)                                                                                                                                            
  File &quot;/usr/lib/python3.6/site-packages/sklearn/pipeline.py&quot;, line 221, in _fit                                                                                                              
    **fit_params_steps[name])                                                                                                                                                                 
  File &quot;/usr/lib/python3.6/site-packages/sklearn/externals/joblib/memory.py&quot;, line 362, in __call__                                                                                           
    return self.func(*args, **kwargs)                                                                                                                                                         
  File &quot;/usr/lib/python3.6/site-packages/sklearn/pipeline.py&quot;, line 588, in _fit_transform_one                                                                                                
    res = transformer.fit_transform(X, y, **fit_params)                                                                                                                                       
  File &quot;/usr/lib/python3.6/site-packages/sklearn/base.py&quot;, line 518, in fit_transform                                                                                                         
    return self.fit(X, **fit_params).transform(X)                                                                                                                                             
  File &quot;/usr/lib/python3.6/site-packages/sklearn/feature_extraction/text.py&quot;, line 519, in transform                                                                                          
    X = self._get_hasher().transform(analyzer(doc) for doc in X)                                                                                                                              
  File &quot;/usr/lib/python3.6/site-packages/sklearn/feature_extraction/hashing.py&quot;, line 160, in transform                                                                                       
    self.alternate_sign)                                                                                                                                                                      
  File &quot;sklearn/feature_extraction/_hashing.pyx&quot;, line 83, in sklearn.feature_extraction._hashing.transform                                                                                   
ValueError: len indices_a: 2532660308, size: -1762306988                                                                                                                                      "><pre class="notranslate"><code class="notranslate">Traceback (most recent call last):                                                                                                                                                            
  File "/usr/lib/python3.6/site-packages/sklearn/pipeline.py", line 289, in fit_transform                                                                                                     
    Xt, fit_params = self._fit(X, y, **fit_params)                                                                                                                                            
  File "/usr/lib/python3.6/site-packages/sklearn/pipeline.py", line 221, in _fit                                                                                                              
    **fit_params_steps[name])                                                                                                                                                                 
  File "/usr/lib/python3.6/site-packages/sklearn/externals/joblib/memory.py", line 362, in __call__                                                                                           
    return self.func(*args, **kwargs)                                                                                                                                                         
  File "/usr/lib/python3.6/site-packages/sklearn/pipeline.py", line 588, in _fit_transform_one                                                                                                
    res = transformer.fit_transform(X, y, **fit_params)                                                                                                                                       
  File "/usr/lib/python3.6/site-packages/sklearn/base.py", line 518, in fit_transform                                                                                                         
    return self.fit(X, **fit_params).transform(X)                                                                                                                                             
  File "/usr/lib/python3.6/site-packages/sklearn/feature_extraction/text.py", line 519, in transform                                                                                          
    X = self._get_hasher().transform(analyzer(doc) for doc in X)                                                                                                                              
  File "/usr/lib/python3.6/site-packages/sklearn/feature_extraction/hashing.py", line 160, in transform                                                                                       
    self.alternate_sign)                                                                                                                                                                      
  File "sklearn/feature_extraction/_hashing.pyx", line 83, in sklearn.feature_extraction._hashing.transform                                                                                   
ValueError: len indices_a: 2532660308, size: -1762306988                                                                                                                                      
</code></pre></div>
<p dir="auto">Proposed fix:</p>
<div class="highlight highlight-source-diff notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="diff --git a/sklearn/feature_extraction/_hashing.pyx b/sklearn/feature_extraction/_hashing.pyx
index e39aeafa0..a3aec8158 100644
--- a/sklearn/feature_extraction/_hashing.pyx
+++ b/sklearn/feature_extraction/_hashing.pyx
@@ -38,7 +38,7 @@ def transform(raw_X, Py_ssize_t n_features, dtype, bint alternate_sign=1):
     # Since Python array does not understand Numpy dtypes, we grow the indices
     # and values arrays ourselves. Use a Py_ssize_t capacity for safety.
     cdef Py_ssize_t capacity = 8192     # arbitrary
-    cdef np.int32_t size = 0
+    cdef np.uint64_t size = 0
     cdef np.ndarray values = np.empty(capacity, dtype=dtype)
 
     for x in raw_X:"><pre class="notranslate"><span class="pl-c1">diff --git a/sklearn/feature_extraction/_hashing.pyx b/sklearn/feature_extraction/_hashing.pyx</span>
index e39aeafa0..a3aec8158 100644
<span class="pl-md">--- a/sklearn/feature_extraction/_hashing.pyx</span>
<span class="pl-mi1">+++ b/sklearn/feature_extraction/_hashing.pyx</span>
<span class="pl-mdr">@@ -38,7 +38,7 @@</span> def transform(raw_X, Py_ssize_t n_features, dtype, bint alternate_sign=1):
     # Since Python array does not understand Numpy dtypes, we grow the indices
     # and values arrays ourselves. Use a Py_ssize_t capacity for safety.
     cdef Py_ssize_t capacity = 8192     # arbitrary
<span class="pl-md"><span class="pl-md">-</span>    cdef np.int32_t size = 0</span>
<span class="pl-mi1"><span class="pl-mi1">+</span>    cdef np.uint64_t size = 0</span>
     cdef np.ndarray values = np.empty(capacity, dtype=dtype)
 
     for x in raw_X:</pre></div>
<h4 dir="auto">Description</h4>
<p dir="auto"><code class="notranslate">sklearn.feature_extraction.text.HashingVectorizer.fit_transform</code> raises <code class="notranslate">ValueError: indices and data should have the same size</code> for data of a certain length. If you chunk the same data it runs fine.</p>
<h4 dir="auto">Steps/Code to Reproduce</h4>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import sklearn
from sklearn.feature_extraction.text import HashingVectorizer
print('scikit-learn version')
print(sklearn.__version__)
vectorizer = HashingVectorizer(
    analyzer='char', non_negative=True,
    n_features=1024, ngram_range=[4,16])
X = ['A'*1432]*203452
print('works')
vectorizer.fit_transform(X[:100000])
print('does not work')
vectorizer.fit_transform(X)"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">sklearn</span>
<span class="pl-k">from</span> <span class="pl-s1">sklearn</span>.<span class="pl-s1">feature_extraction</span>.<span class="pl-s1">text</span> <span class="pl-k">import</span> <span class="pl-v">HashingVectorizer</span>
<span class="pl-en">print</span>(<span class="pl-s">'scikit-learn version'</span>)
<span class="pl-en">print</span>(<span class="pl-s1">sklearn</span>.<span class="pl-s1">__version__</span>)
<span class="pl-s1">vectorizer</span> <span class="pl-c1">=</span> <span class="pl-v">HashingVectorizer</span>(
    <span class="pl-s1">analyzer</span><span class="pl-c1">=</span><span class="pl-s">'char'</span>, <span class="pl-s1">non_negative</span><span class="pl-c1">=</span><span class="pl-c1">True</span>,
    <span class="pl-s1">n_features</span><span class="pl-c1">=</span><span class="pl-c1">1024</span>, <span class="pl-s1">ngram_range</span><span class="pl-c1">=</span>[<span class="pl-c1">4</span>,<span class="pl-c1">16</span>])
<span class="pl-v">X</span> <span class="pl-c1">=</span> [<span class="pl-s">'A'</span><span class="pl-c1">*</span><span class="pl-c1">1432</span>]<span class="pl-c1">*</span><span class="pl-c1">203452</span>
<span class="pl-en">print</span>(<span class="pl-s">'works'</span>)
<span class="pl-s1">vectorizer</span>.<span class="pl-en">fit_transform</span>(<span class="pl-v">X</span>[:<span class="pl-c1">100000</span>])
<span class="pl-en">print</span>(<span class="pl-s">'does not work'</span>)
<span class="pl-s1">vectorizer</span>.<span class="pl-en">fit_transform</span>(<span class="pl-v">X</span>)</pre></div>
<h4 dir="auto">Expected Results</h4>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="scikit-learn version
0.18.1
works
does not work"><pre class="notranslate"><code class="notranslate">scikit-learn version
0.18.1
works
does not work
</code></pre></div>
<h4 dir="auto">Actual Results</h4>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="scikit-learn version
0.18.1
works
does not work
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
&lt;ipython-input-1-aae200adab09&gt; in &lt;module&gt;()
     10 vectorizer.fit_transform(X[:100000])
     11 print('does not work')
---&gt; 12 vectorizer.fit_transform(X)

/Users/benkaehler/miniconda3/envs/qiime2-2017.4/lib/python3.5/site-packages/sklearn/feature_extraction/text.py in transform(self, X, y)
    485 
    486         analyzer = self.build_analyzer()
--&gt; 487         X = self._get_hasher().transform(analyzer(doc) for doc in X)
    488         if self.binary:
    489             X.data.fill(1)

/Users/benkaehler/miniconda3/envs/qiime2-2017.4/lib/python3.5/site-packages/sklearn/feature_extraction/hashing.py in transform(self, raw_X, y)
    147 
    148         X = sp.csr_matrix((values, indices, indptr), dtype=self.dtype,
--&gt; 149                           shape=(n_samples, self.n_features))
    150         X.sum_duplicates()  # also sorts the indices
    151         if self.non_negative:

/Users/benkaehler/miniconda3/envs/qiime2-2017.4/lib/python3.5/site-packages/scipy/sparse/compressed.py in __init__(self, arg1, shape, dtype, copy)
     96             self.data = np.asarray(self.data, dtype=dtype)
     97 
---&gt; 98         self.check_format(full_check=False)
     99 
    100     def getnnz(self, axis=None):

/Users/benkaehler/miniconda3/envs/qiime2-2017.4/lib/python3.5/site-packages/scipy/sparse/compressed.py in check_format(self, full_check)
    165         # check index and data arrays
    166         if (len(self.indices) != len(self.data)):
--&gt; 167             raise ValueError(&quot;indices and data should have the same size&quot;)
    168         if (self.indptr[-1] &gt; len(self.indices)):
    169             raise ValueError(&quot;Last value of index pointer should be less than &quot;

ValueError: indices and data should have the same size"><pre class="notranslate"><code class="notranslate">scikit-learn version
0.18.1
works
does not work
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
&lt;ipython-input-1-aae200adab09&gt; in &lt;module&gt;()
     10 vectorizer.fit_transform(X[:100000])
     11 print('does not work')
---&gt; 12 vectorizer.fit_transform(X)

/Users/benkaehler/miniconda3/envs/qiime2-2017.4/lib/python3.5/site-packages/sklearn/feature_extraction/text.py in transform(self, X, y)
    485 
    486         analyzer = self.build_analyzer()
--&gt; 487         X = self._get_hasher().transform(analyzer(doc) for doc in X)
    488         if self.binary:
    489             X.data.fill(1)

/Users/benkaehler/miniconda3/envs/qiime2-2017.4/lib/python3.5/site-packages/sklearn/feature_extraction/hashing.py in transform(self, raw_X, y)
    147 
    148         X = sp.csr_matrix((values, indices, indptr), dtype=self.dtype,
--&gt; 149                           shape=(n_samples, self.n_features))
    150         X.sum_duplicates()  # also sorts the indices
    151         if self.non_negative:

/Users/benkaehler/miniconda3/envs/qiime2-2017.4/lib/python3.5/site-packages/scipy/sparse/compressed.py in __init__(self, arg1, shape, dtype, copy)
     96             self.data = np.asarray(self.data, dtype=dtype)
     97 
---&gt; 98         self.check_format(full_check=False)
     99 
    100     def getnnz(self, axis=None):

/Users/benkaehler/miniconda3/envs/qiime2-2017.4/lib/python3.5/site-packages/scipy/sparse/compressed.py in check_format(self, full_check)
    165         # check index and data arrays
    166         if (len(self.indices) != len(self.data)):
--&gt; 167             raise ValueError("indices and data should have the same size")
    168         if (self.indptr[-1] &gt; len(self.indices)):
    169             raise ValueError("Last value of index pointer should be less than "

ValueError: indices and data should have the same size
</code></pre></div>
<h4 dir="auto">Versions</h4>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Darwin-16.5.0-x86_64-i386-64bit
Python 3.5.3 |Continuum Analytics, Inc.| (default, Mar  6 2017, 12:15:08) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]
NumPy 1.12.1
SciPy 0.19.0
Scikit-Learn 0.18.1"><pre class="notranslate"><code class="notranslate">Darwin-16.5.0-x86_64-i386-64bit
Python 3.5.3 |Continuum Analytics, Inc.| (default, Mar  6 2017, 12:15:08) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]
NumPy 1.12.1
SciPy 0.19.0
Scikit-Learn 0.18.1
</code></pre></div>
<p dir="auto">and</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Linux-2.6.32-504.16.2.el6.x86_64-x86_64-with-centos-6.6-Final
Python 3.5.3 |Continuum Analytics, Inc.| (default, Mar  6 2017, 11:58:13) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
NumPy 1.12.1
SciPy 0.19.0
Scikit-Learn 0.18.1"><pre class="notranslate"><code class="notranslate">Linux-2.6.32-504.16.2.el6.x86_64-x86_64-with-centos-6.6-Final
Python 3.5.3 |Continuum Analytics, Inc.| (default, Mar  6 2017, 11:58:13) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
NumPy 1.12.1
SciPy 0.19.0
Scikit-Learn 0.18.1
</code></pre></div>