title: Filling preallocated `pandas.DataFrame`
body: <p>I need to append a lot (1 440 000 000) of rows to a <code>pandas.DataFrame</code>.</p>
  
  <p>I know the number of rows in advance, so I can preallocate it, then fill it with data in a C-like manner.</p>
  
  <p>So far the best idea I have is pretty ugly:</p>
  
  <pre><code>&gt;&gt;&gt; N = 1000000
  &gt;&gt;&gt; sham = [-1] * (N * len(THRESHOLDS) * len(OBJECTS)) # 1440000000
  &gt;&gt;&gt; DATA = pd.DataFrame({'threshold': pd.Categorical(sham, categories=THRESHOLDS, ordered=True),
  ...                      'expected': pd.Series(sham, dtype=np.float16),
  ...                      'iteration': pd.Series(sham, dtype=np.int32),
  ...                      'analyser': pd.Categorical(sham, categories=ANALYSERS),
  ...                      'object': pd.Categorical(sham, categories=OBJECTS),
  ...                     },
  ...                     columns=['threshold', 'expected', 'iteration', 'analyser', 'object'])
  &gt;&gt;&gt; ptr = 0
  &gt;&gt;&gt; for t in THRESHOLDS:
  ...     for o in OBJECTS:
  ...         for a in ANALYSERS:
  ...             for i in range(N):
  ...                 DATA.iloc[ptr] = t, expectedMonteCarlo(o, a, t), i, a, o
  ...                 ptr += 1
  </code></pre>
  
  <p>The question is, <strong>how can I make my code cleaner?</strong> I mean especially:</p>
  
  <ul>
  <li>preallocate <code>DATA</code> without inflating it with <code>sham</code> list,</li>
  <li>append rows to preallocated <code>DATA</code> without use of index?</li>
  </ul>
  
  <p>The main problem is memory efficiency. Otherwise I would go for appending records to <code>list</code> object, and then converting it to <code>pandas.DataFrame</code>.</p>