title: How to read a super huge file into numpy array N lines at a time
body: <p>I have a huge file (around 30GB), each line includes coordination of a point on a 2D surface. I need to load the file into Numpy array: <code>points = np.empty((0, 2))</code>, and apply <code>scipy.spatial.ConvexHull</code> over it. Since the size of the file is very large I couldn't load it at once into the memory, I want to load it as batch of N lines and apply <code>scipy.spatial.ConvexHull</code> on the small part and then load the next N rows! What's an efficient to do it?<br>
  I <a href="https://stackoverflow.com/questions/5832856/how-to-read-file-n-lines-at-a-time-in-python">found out</a> that in python you can use <code>islice</code> to read N lines of a file but the problem is <code>lines_gen</code> is a generator object, which gives you each line of the file and should be used in a loop, so I am not sure how can I convert the <code>lines_gen</code> into Numpy array in an efficient way? </p>
  
  <pre><code>from itertools import islice
  with open(input, 'r') as infile:
      lines_gen = islice(infile, N)
  </code></pre>
  
  <p>My input file:</p>
  
  <pre><code>0.989703    1
  0   0
  0.0102975   0
  0.0102975   0
  1   1
  0.989703    1
  1   1
  0   0
  0.0102975   0
  0.989703    1
  0.979405    1
  0   0
  0.020595    0
  0.020595    0
  1   1
  0.979405    1
  1   1
  0   0
  0.020595    0
  0.979405    1
  0.969108    1
  ...
  ...
  ...
  0   0
  0.0308924   0
  0.0308924   0
  1   1
  0.969108    1
  1   1
  0   0
  0.0308924   0
  0.969108    1
  0.95881 1
  0   0
  </code></pre>