Read data with NAs into python and calculate mean row-wise
Question: I am reading in data from a csvfile and attempt to calculate the mean
columnwise. While the number of columns is fixed, the number of rows isn't.
Therefore I first read in the rows I need, make them a list and then form a
numpy array of the list. But it doesn't work.

    
    
    import csv
    import numpy
    

Reading in (loops through every file and find matches, which will then be
appended):

    
    
    with open(input_file, mode='r') as f:
        reader = csv.reader(f, delimiter=';')
        for row in reader:
            pass
            # matchin algorithm omitted
            found_line = row
            del found_line[0] #remove first entry on name
    

`input_file` looks like

    
    
    Weihnachtsmann;16;30.3125;0.00677830307346;0.000491988890358;0.2796728754;0.00371057513915;0.000667111407605;0.00177896375361
    Tannenbaum;6;33.5;0.032918005099;0.00312809941211;0.308224811515;0.0124857679873;0.00644874360685;0.000667111407605
    Heilier Klaus;1;NA;NA;NA;NA;NA;NA;NA
    

Then, I make a list out of the entries that match:

    
    
    author_list.append(','.join(found_line))
    author_array = numpy.array(author_list)
    

I am not creating the numpy array in the first place because I heard it's
unpythonic and slow to append to numpy arrays.

    
    
    print author_arry
    

yields

    
    
    ['1,NA,NA,NA,NA,NA,NA' '6;33.5;0.032918005099;0.00312809941211;0.308224811515;0.0124857679873;0.00644874360685;0.000667111407605' '16;30.3125;0.00677830307346;0.000491988890358;0.2796728754;0.00371057513915;0.000667111407605;0.00177896375361']
    

but I am not even sure if that's an array with the dimensions I want (should
be exactly eight columns) or just one colum and three rows.

Afterwards, I have to convert the `NA`s that come from `R` into numpy's `NaN`
(if I am correctly) and I don't know how to do that. I tried

    
    
    [author_entry.replace('NA','nan') for author_entry in author_list]
    

but I get an error.


Answer: There are a number of different ways you could read in the data from the file
using NumPy. Here's one way using
[`np.genfromtxt`](http://docs.scipy.org/doc/numpy/reference/generated/numpy.genfromtxt.html).
The names in the first column become NumPy `nan` values, as do any other non-
float strings in your file:

    
    
    >>> arr = np.genfromtxt(input_file, delimiter=';', dtype=np.float64)
    >>> arr
    array([[             nan,   1.60000000e+01,   3.03125000e+01,
              6.77830307e-03,   4.91988890e-04,   2.79672875e-01,
              3.71057514e-03,   6.67111408e-04,   1.77896375e-03],
           [             nan,   6.00000000e+00,   3.35000000e+01,
              3.29180051e-02,   3.12809941e-03,   3.08224812e-01,
              1.24857680e-02,   6.44874361e-03,   6.67111408e-04],
           [             nan,   1.00000000e+00,              nan,
                         nan,              nan,              nan,
                         nan,              nan,              nan]])
    

This is an array with 3 rows and 9 columns. To remove the first entry on each
line, you could just slice and reassign with `arr = arr[:, 1:]`.

You can calculate the row-wise mean using `np.nanmean` (to ignore the `nan`
values when calculating the mean):

    
    
    >>> np.nanmean(arr, axis=1)
    array([ 5.82569998,  4.98298407,  1.        ])
    

