Reading a big file cost too much memory in Python 2.7
<p>I used .readline() to parse file line by line, because I need
to find out the start position to extract data into a list, and the end
point to pause extracting, then repeat until the end of file.
My file to read is formatted like this:</p>

<pre>
blabla...
useless....
...
/sign/
data block(e.g. 10 cols x 1000 rows) 
... blank line 
/sign/    
data block(e.g. 10 cols x 1000 rows)
... blank line 
... 
EOF
</pre>

<p>let's call this file 'myfile'
and my python snippet:</p>

<pre><code>f=open('myfile','r')
blocknum=0 #number the data block
data=[]
while True:
       # find the extract begnning
       while not f.readline().startswith('/sign/'):pass
       # creat multidimensional list to store data block
       data=append([])
       blocknum +=1
       line=f.readline()

       while line.strip():
       # check if the line is a blank line, i.e the end of one block
               data[blocknum-1].append(["2.6E" %float(x) for x in line.split()])
               line = f.readline()
       print "Read Block %d" %blocknum
       if not f.readline(): break
</code></pre>

<p>The running result was that read a 500M file consume almost 2GB RAM, I
cannot figure it out, somebody help!
Thanks very much!</p>
