Efficiently rewriting lines in a large text file with Python
<p>I'm trying to generate a large data file (in the GBs) by iterating over thousands of database records. At the top of the file are a line for each "feature" that appears latter in the file. They look like:</p>

<pre><code>@attribute 'Diameter' numeric
@attribute 'Length' real
@attribute 'Qty' integer
</code></pre>

<p>lines containing data using these attributes look like:</p>

<pre><code>{0 0.86, 1 0.98, 2 7}
</code></pre>

<p>However, since my data is sparse data, each record from my database may not have each attribute, and I don't know what the complete feature set is in advance. I could, in theory, iterate over my database records twice, the first time accumulating the feature set, and then the second time to output my records, but I'm trying to find a more efficient method.</p>

<p>I'd like to try a method like the following pseudo-code:</p>

<pre><code>fout = open('output.dat', 'w')
known_features = set()
for records in records:
    if record has unknown features:
        jump to top of file
        delete existing "@attribute" lines and write new lines
        jump to bottom of file
    fout.write(record)
</code></pre>

<p>It's the jump-to/write/jump-back part I'm not sure how to pull off. How would you do this in Python?</p>

<p>I tried something like:</p>

<pre><code>fout.seek(0)
for new_attribute in new_attributes:
    fout.write(attribute)
fout.seek(0, 2)
</code></pre>

<p>but this overwrites both the attribute lines <em>and</em> data lines at the top of the file, not simply insert new lines starting at the seek position I specify.</p>

<p>How do you obtain a word-processor's "insert" functionality in Python <strong>without</strong> loading the entire document into memory? The final file is larger than all my available memory.</p>
