Python MemoryError when storing saved lists
Question: I am new to python, so I apologize if this example is trivial.

I am trying to write a simple script that will pase and extract parts of two
large datafiles (~40gb each) into one resulting file with a slightly altered
format. I originally tried to use readlines(), but that reads all of the files
into memory, and our instance only has 28gb of memory. Using the sizehint
parameter only parses a portion of the file.

I am now iterating over the file. The problem is that I store the output of
the text parsing in three lists that grow to be rather large, eclipsing the
available memory. I thought this would just switch to using the swap, which
would be fine, but it instead just exits with a "MemoryError".

This works fine with small sample files, but chokes on our actual data.

The script:

    
    
    import sys
    
    a = []
    b = []
    c = []
    
    file1 = open(sys.argv[1],"r")
    for line in file1:
        if '@' in line:
            a.append(line.lstrip('@').rstrip('\n'))
            b.append(file1.next().rstrip('\n'))
    file1.close()
    
    file2 = open(sys.argv[2],"r")
    for line in file2:
        if '@' in line: 
            c.append(file2.next().rstrip('\n'))
    file2.close()
    
    file3 = open(sys.argv[3],"w")
    for i in xrange(len(a)):
        file3.write("".join([">",a[i],'\n',b[i],":",c[i],"\n"]))
    

What I have found online suggests creating some sort of database to store the
variables, but that shouldn't be required. Do you have any ideas how I should
deal with this?

For completeness, this is what I'm trying to do (from our example test-data:

    
    
    file1: 
    
    @Read.Salmonella_paratyphi_A_chromosome.29004.4835/1
    TCGTGTACAGCATTCTTTATAGTGGAACGGTGACCGTACCGCAAAGCTGCGAAATCAACGCCGGACKIPPTCGTAG
    +
    AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
    
    file2:
    
    @Read.Salmonella_paratyphi_A_chromosome.29004.4835/1
    TCGTGTACAGCATTCTTTATAGTGGAACGGTGACCGTACCGCAAAGCTGCGAAATCAACGCCGGACAAACGATTCT
    +
    AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
    
    file3 (output):
    
    >Read.Salmonella_paratyphi_A_chromosome.29004.4835/1
    TCGTGTACAGCATTCTTTATAGTGGAACGGTGACCGTACCGCAAAGCTGCGAAATCAACGCCGGACKIPPTCGTAG:TCGTGTACAGCATTCTTTATAGTGGAACGGTGACCGTACCGCAAAGCTGCGAAATCAACGCCGGACAAACGATTCT
    


Answer: Instead of parsing the files into the arrays (`a`, `b` and `c`), can you write
into the file as you parse the files?

Something like this pseudo'ish code:

    
    
    def get_line_with_at(a):
         while a:
             line = a.readline()
             if "@" in line:
                 return line.strip()
    
    
    # Open all file handles
    a, b, c = [open(sys.argv[x + 1]) for x in range(3)]
    out = open(sys.argv[4])
    
    while a and b and c:
        # Repeat until a, b, and file handles are exhausted
        chunk1 = get_line_with_at(a)
        chunk2 = b.next().strip()
        chunk3 = get_line_with_at(c)
    
         out.write(
             ">%s\n%s:%s\n" % (chunk1, chunk2, chunk3))
    

That way you should only have to load very little into memory (theoretically 4
file handles and the contents of the current lines)

