Reading multiple data files with a loop on Python
Question: Hi I have a over 200 data files with name similar names like
abc.20.0000.catalog.out , abc.20.1000.catalog.out, abc.20.2000.catalog.out
...... abc.40.0000.catalog.out

Each file contains data like this

    
    
       Group catalog for redshift  18.1000
     1) group ID
     2) group mass (Msun/h)
     3- 5) initial position (Mpc/h)
     6- 8) final position (Mpc/h)
     9-11) velocity (km/s)
     12) number of particles  
    
    250103187  0.227591E+08 1.86  1.03  2.51  1.65  1.06  2.53  -47.56  7.50  3.83    328
    202456030  0.167918E+08 0.29  4.57  2.02  0.23  4.63  2.14  -13.27  10.67 3.68    242
    89479147  0.763262E+06  1.47  4.80  0.89  1.34  4.83  0.99  -28.90  6.20  17.30    11 
    

each such file contains over 10^6 lines.

I want to do the following: 1\. I want to read the data from each file and
erase the text on top. 2\. I want to then store the data from all these files
into a single big list of matrices, each matrix being the data from each of
these files.


Answer: Here is a Python/Pandas solution:

    
    
    import pandas as pd
    import glob
    
    L = []
    for f in glob.glob('abc*'):       
       df = pd.read_csv(f,skiprows=1)
       L.append(df.values)
    

