Python multiprocessing with arrays and multiple arguments
Question: So I am trying to read in a bunch of very large data files and each one takes
quite some time to load. I am trying to figure out how to load them in the
quickest way and without running into memory problems. Once the data files are
loaded into the array the correct way I do not need to write to them, but just
need to read. I've been trying to parallelize this for some time, but can't
figure it out.

Let's say I have 400 time files. Each of these files is tab separated and has
30 variables each with 40,000 data points. I would like to create a
400x30x40000 array so that I can access the points easily. The data file is
set up so that the first 40k points is for variable 1, the second 40k is for
var 2, and so on.

I have written a function that loads in a time file correctly and stores it in
my array correctly. What I'm having trouble with is parallelizing it. This
does work if I put it in a for loop and iterate over i.

    
    
    import h5py
    import pandas as pd
    h5file = h5py.File('data.h5','a')
    data = h5file.create_dataset("default",(len(files),len(header),numPts))
           # is shape 400x30x40000
    def loadTimes(files,i,header,numPts,data):
       # files has 400 elements
       # header has 30 elements
       # numPts is an integer
       allData = pd.read_csv(files[i],delimiter="\t",skiprows=2,header=None).T
       for j in range(0,len(header)):
          data[i,j,:] = allData[0][j*numPts:(j+1)*numPts]
       del allData
    

files is the list of time files loaded by `subprocess.check_output` (has about
400 elements), header is the list of variables, loaded from another file (has
30 elements in it). numPts is the number of points per variable (so around
40k).

I've tried using `pool.map` to load the data but found it didn't like multiple
arguments. I also tried using partial, zip, and the lambda function, but none
of those seem to like my arrays.

I am not set in stone about this method. If there is a better way to do it I
will greatly appreciate it. It will just take too long to load in all this
data one at a time. My calculations show that it would take ~3hrs to load on
my computer using one core. And I will use up A LOT of my memory. I have
access to another machine with a lot more cores, which is actually where I
will be doing this, and I'd like to utilize them properly.


Answer: So how I solved this was using the h5 file format. What I did was write the
loops so that they only had the iter

    
    
    def LoadTimeFiles(i):
       from pandas import read_csv
       import h5py as h5
       dataFile = h5.File('data.h5','r+')
       rFile = dataFile['files'][i]
       data = dataFile['data']
       lheader = len(data[0,:,0])
       numPts = len(data[0,0,:])
       allData = read_csv(rFile,delimiter="\t",skiprows=2,header=None,low_memory=False).T
       for j in range(0,lheader):
          data[i,j,:] = allData[0][j*numPts:(j+1)*numPts]
       del allData
       dataFile.close()
    
    def LoadTimeFilesParallel(np):
       from multiprocessing import Pool, freeze_support
       import h5py as h5
       files = h5.File('data.h5','r')
       numFiles = len(files['data'][:,0,0])
       files.close()
       pool = Pool(np)
       freeze_support
       pool.map(LoadTimeFiles,range(numFiles))
    
    if __name__ == '__main__':
       np = 5
       LoadTimeFilesParallel(np)
    

So since I was storing the data in the h5 format anyway I thought I'd be
tricky and load it up in each loop (I can see no time delays in reading the h5
files). I added the option `low_memory=False` to the `read_csv` command
because it made it go faster. The j loop was really fast so I didn't need to
speed it up.

Now each `LoadTimeFile` loop takes about 20-30 secs and we do 5 at once
without order mattering. My ram never hits above 3.5Gb (total system usage)
and drops back to under a gig after runs.

