python script slow read and write gz files
Question: I have a xxx.wig.gz file, that have 3,000,000,000 lines in such format:

    
    
    fixedStep chrom=chr1 start=1 step=1
    0
    0
    0
    0
    0
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    ...
    fixedStep chrom=chr2 start=1 step=1
    0
    0
    0
    0
    0
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    ...
    

and i want to

  1. break it down by "chrom". So every time I read a line starts with "fixedstep", I create a new file and close old one.
  2. I want 0/1 output by comparing each value to a "threshold", pass=1 otherwise 0

below is my python script which runs super slow (I am projecting it to finish
~10hours, so far 2 chromosomes done after ~1 hour)

can someone help me improve it?

    
    
    #!/bin/env python    
    import gzip    
    import re    
    import os    
    import sys   
    
    fn = sys.argv[1]    
    f  = gzip.open(fn)    
    fo_base = os.path.basename(fn).rstrip('.wig').rstrip('.wig.gz')    
    fo_ext  = '.bt.gz'    
    thres = 100    
    fo = None    
    for l in f:    
        if l.startswith("fixedStep"):    
            if fo is not None:    
                fo.flush()    
                fo.close()    
            fon  = re.search(r'chrom=(\w*)', l).group(0).split('=')[-1]    
            fo   = gzip.open(fo_base + "_" + fon + fo_ext,'wb')    
        else:    
            if int(l.strip())>= thres:    
                fo.write("1\n")    
            else:    
                fo.write("0\n")    
    if fo is not None:    
        fo.flush()    
        fo.close()    
    f.close() 
    

PS. I assume awk can do it much faster but I am not great with awk


Answer: Thanks Summer for editing the text.  
I added buffered read/write to the script and now it is several times faster
(still relatively slow though):

    
    
    import io    
    f  = io.BufferedReader( gzip.open(fn) )    
    fo   = io.BufferedWriter( gzip.open(fo_base + "." + fon + fo_ext,'wb') )    
    

