Use readlines() with indices or parse lines on the fly?
<p>I'm making a simple test function that asserts that the output from an interpreter I'm developing is correct, by reading from a file the expression to evaluate and the expected result, much like python's doctest. This is for scheme, so an example of an input file would be</p>

<pre><code>&gt; 42
42

&gt; (+ 1 2 3)
6
</code></pre>

<p>My first attempt for a function that can parse such a file looks like the following, and it seems to work as expected:</p>

<pre><code>def run_test(filename):
    interp = Interpreter()
    response_next = False
    num_tests = 0
    with open(filename) as f:
        for line in f:
            if response_next:
                assert response == line.rstrip('\n')
                response_next = False
            elif line.startswith('&gt; '):
                num_tests += 1
                response = interp.eval(line[2:])
                response = str(response) if response else ''
                response_next = True
    print "{:20} Ran {} tests successfully".format(os.path.basename(filename),
                                                    num_tests)
</code></pre>

<p>I wanted to improve it slightly by removing the <code>response_next</code> flag, as I am not a fan of such flags, and instead read in the next line within the <code>elif</code> block with <code>next(f)</code>. I had a small unrelated question regarding that which I asked about in IRC at freenode. I got the help I wanted but I was also given the suggestion to use <code>f.readlines()</code> instead, and then use indexing on the resulting list. (I was also told that I could use <code>groupby()</code> in <code>itertools</code> for the pairwise lines, but I'll investigate that approach later.)</p>

<p>Now to the question, I was very curious why that approach would be better, but my Internet connection was a flaky one on a train and I was unable to ask, so I'll ask it here instead. Why would it be better to read everything with <code>readlines()</code> instead of parsing every line as they are read on the fly?</p>

<p>I'm really wondering as my feeling is the opposite, I think it seems cleaner to parse the lines one at a time so that everything is finished in one go. I usually avoid using indices in arrays in Python and prefer to work with iterators and generators. Maybe it is impossible to answer and guess what the person was thinking in case it was a subjective opinion, but if there is some general recommendation I'd be happy to hear about it.</p>
