Using generator for buffered read of large file in Python I have a large file that I need to parse - and since it will be regenerated from external queries every time script runs so there is no way to parse it once and cache the results.

I want to save memory footprint and read and parse only logical "chunks" of that file everything between open 'product' and closing curly bracket.
Not sure what is the canonical way in Python (I am new to the language). 

Here's what I tried so far:

def read_chunk(file_name, pattern_open_line, pattern_close_line):
    with open(file_name,"r") as in_file:
        chunk = []
        in_chunk = False
        open_line = re.compile(pattern_open_line);
        close_line = re.compile(pattern_close_line)
        try:
            for line in in_file:
                line = line.strip()
                if in_chunk:
                    chunk.append(line)
                if close_line.match(line):
                    yield chunk
                if open_line.match(line):
                    chunk = []
                    chunk.append(line)
                    in_chunk = True
                    continue
        except StopIteration:
            pass

def get_products_buffered(infile):
    chunks = read_chunk(infile, r'^product\s*$', r'^\s*\}\s*')
    products = []
    for lines in chunks:
        for line in lines:
            if line.startswith('productNumber:'):
                 productNumber = line[len('productNumber:'):].strip().rstrip(';').strip('"')
                 products.append(productNumber)
                 continue
    return products

def get_products_unbuffered(infile):
    with open(infile) as f:
        lines = f.readlines()
        f.close()
    products = []
    for line in lines:
        if line.startswith('productNumber:'):
             productNumber = line[len('productNumber:'):].strip().rstrip(';').strip('"')
             products.append(productNumber)
             continue
    return products


I profiled both runs and while unbuffered reading is faster:

Buffered reading
Found 9370 products:
Execution time: 3.0031037185720177
Unbuffered reading
Found 9370 products:
Execution time: 1.2247122452647523


it also incurs a much bigger memory hit when file is essentially read into memory:

Line #    Mem usage    Increment   Line Contents
================================================
    29     28.2 MiB      0.0 MiB   @profile
    30                             def get_products_buffered(infile):
    31     28.2 MiB      0.0 MiB       chunks = read_chunk(infile, '^product\s*$', '^\s*\}\s*')
    32     28.2 MiB      0.0 MiB       products = []
    33     30.1 MiB      1.9 MiB       for lines in chunks:


versus:

Line #    Mem usage    Increment   Line Contents
================================================
    42     29.2 MiB      0.0 MiB   @profile
    43                             def get_products_unbuffered(infile):
    44     29.2 MiB      0.0 MiB       with open(infile) as f:
    45    214.5 MiB    185.2 MiB           lines = f.readlines()


I would be grateful for any pointers/suggestions.
You called it unbuffered, but these lines:

with open(infile) as f:
    lines = f.readlines()
    f.close()


slurp the entire file into memory, while your 'buffered' version only pulls in a line at a time, returning chunks.

I note that you're not doing anything with the entire chunk, just the lines starting with 'productNumber:', so I think a rework of your 'unbuffered' code will actually be fastest, as well as clearest:

def get_products_unbuffered(infile):
    products = []
    with open(infile) as f:
        for line in f:
            if line.startswith('productNumber:'):
                 productNumber = line[len('productNumber:'):].strip().rstrip(';').strip('"')
                 products.append(productNumber)
    return products


as this will read the file a line at a time and only keep desired info (productNumbers) in memory..