Load huge data to memory using python
Question: I have to load large data to the memory, and the structure is list. How can I
get another approach. thanx


Answer: process the data line by line, eg

    
    
    for line in open("file"):
        print "do processing of line:",line
    

Or if you really really want to load one whole chunk to memory, you can try
memory mapping.

    
    
    import mmap,os,re
    bigfile = open( "file2" )
    length = os.stat(bigfile.fileno()).st_size
    try:
        mapping = mmap.mmap(bigfile.fileno(), length, mmap.MAP_PRIVATE, mmap.PROT_READ )
    except AttributeError:
        mapping = mmap.mmap(bigfile.fileno(), 0, None, mmap.ACCESS_READ )
    data = mapping.read(length)
    # example, finding a pattern
    pat =re.compile("pattern",re.M|re.DOTALL)
    print pat.findall(data)
    

