title: Create a pandas DataFrame from generator?
body: <p>I've create a tuple generator that extract information from a file filtering only the records of interest and converting it to a tuple that generator returns.</p>
  
  <p>I've try to create a DataFrame from:</p>
  
  <pre><code>import pandas as pd
  df = pd.DataFrame.from_records(tuple_generator, columns = tuple_fields_name_list)
  </code></pre>
  
  <p>but throws an error:</p>
  
  <pre><code>... 
  C:\Anaconda\envs\py33\lib\site-packages\pandas\core\frame.py in from_records(cls, data, index, exclude, columns, coerce_float, nrows)
     1046                 values.append(row)
     1047                 i += 1
  -&gt; 1048                 if i &gt;= nrows:
     1049                     break
     1050 
  
  TypeError: unorderable types: int() &gt;= NoneType()
  </code></pre>
  
  <p>I managed it to work consuming the generator in a list, but uses twice memory:</p>
  
  <pre><code>df = pd.DataFrame.from_records(list(tuple_generator), columns = tuple_fields_name_list)
  </code></pre>
  
  <p>The files I want to load are big, and memory consumption matters. The last try my computer spends two hours trying to increment virtual memory :(</p>
  
  <p><strong>The question:</strong> Anyone knows a method to create a DataFrame from a record generator directly, without previously convert it to a list?</p>
  
  <p>Note: I'm using python 3.3 and pandas 0.12 with Anaconda on Windows.</p>
  
  <p><strong>Update:</strong></p>
  
  <p>It's not problem of reading the file, my tuple generator do it well, it scan a text compressed file of intermixed records line by line and convert only the wanted data to the correct types, then it yields fields in a generator of tuples form.
  Some numbers, it scans 2111412 records on a 130MB gzip file, about 6.5GB uncompressed, in about a minute and with little memory used.</p>
  
  <p>Pandas 0.12 does not allow generators, dev version allows it but put all the generator in a list and then convert to a frame. It's not efficient but it's something that have to deal internally pandas. Meanwhile I've must think about buy some more memory.</p>