title: Creating large .npy files with memory efficiency
body: <p>I'm trying to create very large .npy files and I'm having a bit of difficulty. For instance, I need to create a (500, 1586, 2048, 3) matrix and save it to a npy file. And preferably, I need to put it in an npz_compressed file. I also need this to be memory efficient to be ran on low-memory systems. I've tried a few methods, but none have seemed to work so far. I've written and re-written things so many times, that I don't have code snippets for everything, but I'll describe the methods as best I can with code snippets where I can. Also, apologies for bad formatting.</p>
  <ol>
  <li>Create an ndarray with all my data in it, then use savez_compressed to export it.</li>
  </ol>
  <p>This gets all my data into the array, but it's terrible for memory efficiency. I filled all 8gb of RAM, plus 5gb of Swap space. I got it to save my file, but it doesn't scale, as my matrix could get significantly larger.</p>
  <ol start="2">
  <li>Use &quot; np.memmap('file_name_to_create', mode='w+', shape=(500,1586,2048,3)) &quot; to create the large, initial npy file, then add my data.</li>
  </ol>
  <p>This method worked for getting my data in, and it's pretty memory efficient. However, i can no longer use np.load to open the file (get errors associated with pickle, regardless of if allow_pickle is true or false), which means I can't put it into compressed. I'd be happy with this format, if I can get it into the compressed format, but I just can't figure it out. I'm trying to avoid using gzip if possible.</p>
  <ol start="3">
  <li>Create a (1,1,1,1) zeros array and save it with np.save. Then try opening it with np.memmap with the same size as before.</li>
  </ol>
  <p>This runs into the same issues as method 2. Can no longer use np.load to read it in afterwards</p>
  <ol start="4">
  <li>Create 5 [100,...] npy files with method 1, and saving them with np.save. Then read 2 in using np.load(mmap_mode='r+') and then merge them into 1 large npy file.</li>
  </ol>
  <p>Creating the individual npy files wan't bad on memory, maybe 1gb to 1.5gb. However, I couldn't figure out how to then merge the npy files without actually loading the entire npy file into RAM. I read in other stackoverflow that npy files aren't really designed for this at all. They mentioned it would be better to use a .h5 file for doing this kind of 'appending'.</p>
  <p>Those are the main methods that I've used. I'm looking for feedback on if any of these methods would work, which one would work 'best' for memory efficiency, and maybe some guidance on getting that method to work. I also wouldn't be opposed to moving to .h5 if that would be the best method, I just haven't tried it yet.</p>