title: How to load a big numpy array from a text file with Dask?
body: <p>I have a text file containing data that I read to memory with <a href="https://numpy.org/doc/stable/reference/generated/numpy.genfromtxt.html" rel="nofollow noreferrer"><code>numpy.genfromtxt</code></a> enforcing a custom <code>numpy.dtype</code>. Although the text file is smaller than the available RAM, I often get a <code>MemoryError</code> (which I don't understand, but it is not the point of this question). When looking for ways to resolve it, I came across <a href="https://docs.dask.org/en/latest/" rel="nofollow noreferrer">dask</a>. In the API I found <a href="https://docs.dask.org/en/latest/array-api.html" rel="nofollow noreferrer">methods</a> for data loading but none of them reads from text files, not to mention my need to support converters in <code>genfromtxt()</code>.</p>
  <p>I see there is a <code>dask.dataframe.read_csv()</code> method, but in my case I don't use pandas, but rather plain <code>numpy.array</code> with custom dtypes and colum names, as mentioned above. The text file I have is not CSV anyway (thus the abovementioned use of converters in <code>genfromtxt()</code>).</p>
  <p>Any ideas on how could I handle it will be appreciated.</p>