Why doesn't np.genfromtxt() remove header while importing in Python?
Question: I have data of the form:

    
    
    #---------------------
    # Data
    #---------------------
    p   q   r   y 1 y 2 y 3 y 4
    2   8   14  748 748 748 790
    2   9   22  262 245 252 328
    1   5   19  512 514 511 569
    2   7   19  748 748 748 805
    3   11  13  160 168 108 164
    2   7   20  788 788 788 848
    1   4   15  310 310 310 355
    3   12  17  230 210 213 218
    

And I am trying to generate array B by using np.genfromtxt(), using the code:

    
    
    import numpy as np
    A = open('data.dat', "r")
    line = A.readline()
    while line.startswith('#'):
        line = A.readline()
    A_header = line.split("\t")
    A_header[-1] = A_header[-1].strip()
    B = np.genfromtxt('data.dat', comments='#', delimiter='\t', names = A_header, dtype = None, unpack = True).transpose()
    print B
    print B['y_1']
    

I have two questions:

  1. _Why doesn't np.genfromtxt() remove data header while importing? When data is imported array B still has the header p, q, ... y 3, y 4._

  2. _Why do we have to provide underscore for header names, e.g. y_1, y_2, etc.? Why can't we provide names as it is y 1, y 2 ... y 4?_


Answer: Instead of opening the file twice, what about:

    
    
    import numpy as np
    
    with open('input.txt', "r") as data:
        while True:
            line = data.readline()
            if not line.startswith('#'): break
    
        header = [e for e in line.strip().split('\t') if e]
        print(header)
    
        B = np.genfromtxt(data, names=header, dtype=None, delimiter='\t')
    
    print B
    print B['y_1']
    

Output:

    
    
    # header
    ['p', 'q', 'r', 'y 1', 'y 2', 'y 3', 'y 4']
    
    # B
    [(2, 8, 14, 748, 748, 748, 790) (2, 9, 22, 262, 245, 252, 328)
     (1, 5, 19, 512, 514, 511, 569) (2, 7, 19, 748, 748, 748, 805)
     (3, 11, 13, 160, 168, 108, 164) (2, 7, 20, 788, 788, 788, 848)
     (1, 4, 15, 310, 310, 310, 355) (3, 12, 17, 230, 210, 213, 218)]
    
    # B['y_1']
    [748 262 512 748 160 788 310 230]
    

Instead of passing the filename to `np.genfromtxt`, here, you pass `data` the
file reader generator.

Otherwise, you get into a weird situation where `skip_header` doesn't really
work, because it considers comment lines. So you'd have to say `skip_header=4`
(3 comment lines + 1 header line) when what makes is `skip_header=1`.

So this approach first "throws out" comment lines. Then for the next line it
extracts the headers. It passes the remaining lines into the `np.genfromtxt`
function with the associated headers.

A few notes:

  * `unpack=True` \+ `transpose()` cancel each other out. So the effect of using both is the same as using neither. So use neither.

  * And if you really want to access the fields using names with spaces (instead of underscores) you can always rename the fields after you generate the `ndarray`:
    
        B.dtype.names = [n.replace('_', ' ') for n in B.dtype.names]
    print B['y 1']  # [748 262 512 748 160 788 310 230]
    

