title: python sklearn read very big svmlight file
body: <p>I am using python 2.7 with svmlight to store and read a very big svmlight format file.</p>
  
  <p>I am reading the file using  </p>
  
  <pre><code>import sklearn
  rows, labels = sklearn.datasets.load_svmlight_file(matrixPath, zero_based=True)
  </code></pre>
  
  <p>The file is too big to be stored in memory. I am looking for a way to iterate over the file in batches without the need to split the file in advance. </p>
  
  <p>For now the best way i found is to split the svmlight file using terminal command split. and then reading the partial files i created. </p>
  
  <p>I <a href="https://stackoverflow.com/questions/6475328/read-large-text-files-in-python-line-by-line-without-loading-it-in-to-memory">found</a> that a good way to read big files is reading in batches of line by line in order not to overflow the memory. </p>
  
  <p>How can i do this with svmlight formated files? </p>
  
  <p>Thanks! </p>