<h4 dir="auto">Description</h4>

<p dir="auto">Please refer to the link below<br>
<a href="https://stackoverflow.com/questions/49848932/sklearn-load-svmlight-file-overflowerror-python-int-too-large-to-convert-to-c-lo" rel="nofollow">https://stackoverflow.com/questions/49848932/sklearn-load-svmlight-file-overflowerror-python-int-too-large-to-convert-to-c-lo</a></p>
<p dir="auto">My data format is libsvm below. " label &lt;feature_index&gt;:&lt;feature_value&gt;". The data type of feature_index is uint64. I got the error here</p>
<p dir="auto">File "sklearn/datasets/_svmlight_format.pyx", line 97, in sklearn.datasets._svmlight_format._load_svmlight_file<br>
OverflowError: Python int too large to convert to C long</p>
<h4 dir="auto">Steps/Code to Reproduce</h4>

<p dir="auto">X_all, y_all = load_svmlight_file(libsvm_filename, dtype=np.float64)<br>
or<br>
X_all, y_all = load_svmlight_file(libsvm_filename, dtype=np.uint64)</p>
<h4 dir="auto">Expected Results</h4>

<h4 dir="auto">Actual Results</h4>

<p dir="auto">Fail to load</p>
<h4 dir="auto">Versions</h4>

<blockquote>
<blockquote>
<blockquote>
<p dir="auto">import platform; print(platform.platform())<br>
Linux-3.10.106-1-tlinux2-0044-x86_64-with-centos-7.2-Final<br>
import sys; print("Python", sys.version)<br>
Python 3.5.4 (default, Apr 11 2018, 20:14:45)<br>
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)]<br>
import numpy; print("NumPy", numpy.<strong>version</strong>)<br>
NumPy 1.14.2<br>
import scipy; print("SciPy", scipy.<strong>version</strong>)<br>
SciPy 1.0.1<br>
import sklearn; print("Scikit-Learn", sklearn.<strong>version</strong>)<br>
Scikit-Learn 0.19.1</p>
</blockquote>
</blockquote>
</blockquote>
<h4 dir="auto">Description</h4>
<p dir="auto">I am using scikit-learn (0.19.1) in the following setup: Ubuntu 17.10 (x86_64) with numpy-1.14.1 and scipy-1.0.0</p>
<p dir="auto">When I tried loading a big libsvm file (1696462087 rows) like this:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from sklearn.datasets import load_svmlight_file
X_train, y_train = load_svmlight_file('/home/user_name/sets/big_dataset.txt')"><pre class="notranslate"><code class="notranslate">from sklearn.datasets import load_svmlight_file
X_train, y_train = load_svmlight_file('/home/user_name/sets/big_dataset.txt')
</code></pre></div>
<p dir="auto">I got the following error:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="...
File &quot;sklearn/datasets/_svmlight_format.pyx&quot;, line 114, in sklearn.datasets._svmlight_format._load_svmlight_file

OverflowError: signed integer is greater than maximum"><pre class="notranslate"><code class="notranslate">...
File "sklearn/datasets/_svmlight_format.pyx", line 114, in sklearn.datasets._svmlight_format._load_svmlight_file

OverflowError: signed integer is greater than maximum
</code></pre></div>
<p dir="auto">I googled around and found a similar <a href="https://github.com/scikit-learn/scikit-learn/issues/5269" data-hovercard-type="issue" data-hovercard-url="/scikit-learn/scikit-learn/issues/5269/hovercard">issue </a>which, however, was related to the <code class="notranslate">query_id</code> parameter. By looking at the <code class="notranslate">_svmlight_format.pyx</code> <a href="https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/datasets/_svmlight_format.pyx">function </a> I can see that both <code class="notranslate">indices </code>and <code class="notranslate">indptr </code> arrays are of type interger and the overflow error occurs when the following assignment is attempted:</p>
<p dir="auto"><code class="notranslate">indptr[len(indptr) - 1] = len(data)</code></p>
<p dir="auto">Any remedy for such a problem? Would the <a href="https://github.com/olologin/scikit-learn/commit/7f0682ac22987c2875334c19a4916aa3a0b1f446">fix</a> be the same as in the previous issue?</p>
<p dir="auto">P.S. I would like to avoid splitting a big libsvm file into many small chunks.</p>