Huge file handling and sorting using python
Question: Im currently working on a program that uses a file having data in the format -
6 columns and dynamic no. of rows.

The file I got for testing is 26 mb and following is the program that converts
first 3 columns into 3 different lists.

    
    
    f = open('foo', 'r')
    print('running...')
    a = []
    b = []
    c = []
    for line in f:
        x = (line.split(' '))
        a.append(x[0])
        b.append(x[1])
        c.append(x[2])
    print(a,b,c,sep='\n')
    

I have rechecked this program and logic looks correct and when implemented on
small file it works but when i use this program with the 26 mb file it stops
responding.

Description of the program: The program opens a file name 'foo' and implements
line by line of the file. It splits the line into parts based on the separator
that is defined as an argument in the .split() method. In my program I have
used white space as an separator as in the text file the data is separated
using white spaces.

Im not able to figure out why this program stops responding and I need help
with it!


Answer: if you use `numpy`, you can use `genfromtxt`:

    
    
    import numpy as np
    
    a,b,c=np.genfromtxt('foo',usecols=[0,1,2],unpack=True)
    

Does that work with your large file?

EDIT:

OK, so I tried it on your file, and it seems to work fine. So I'm not sure
what your problem is.

    
    
    In [1]: from numpy import genfromtxt
    
    In [2]: a,b,c=genfromtxt('foo',usecols=[0,1,2],unpack=True)
    
    In [3]: a
    Out[3]: 
    array([ 406.954744,  406.828508,  406.906079, ...,  408.944226,
            408.833872,  408.788698])
    
    In [4]: b
    Out[4]: 
    array([ 261.445358,  261.454366,  261.602131, ...,  260.46189 ,
            260.252377,  260.650606])
    
    In [5]: c
    Out[5]: 
    array([ 17.451789,  17.582017,  17.388673, ...,  26.41099 ,  26.481148,
            26.606282])
    
    In [6]: print len(a), len(b), len(c)
    419040 419040 419040
    

