How to preallocate memory when creating pandas DataFrames
Use approach 1, but every time you get to (say) 10000 rows, convert the list of lists to a DataFrame, append the dataframe to a list of dataframes, and clear your list of lists (by setting it to []). At the end, take your list of df chunks and pd.concat them together.
During the execution of my job I want to build a pandas DataFrame with its internal state at each step of execution. I want to do that because I want to be able to use the DataFrame later to do some analysis. So, that's the context: I start with an empty DataFrame and, after the job has finishing producing all its data, the final DataFrame size is 7GB (at most).
These are the approaches I've tried this far:

I can maintain a list of lists, my job keeps appending new rows to that list. At the end I run something like: df = pd.DataFrame(rows, columns=list("abc")). This is the fastest method, but it's very wasteful in terms of memory usage.
I don't use the lists of lists anymore, but, instead I keep calling df = df.append([{"a": 1.0, "b": 1.0, "c": "hello"}]). This starts off ok, but as the DataFrame grows a lot of memory copying starts happening and it becomes really slow.

So, neither of them are ideal. I can estimate the final size of the DataFrame before my job runs (I think I can usually match the size precisely, to be honest), so I would like to tell pandas to allocate the entire memory ahead (or at least a large enough chunk to last for a while) and then, I'd like a method similar to df.append that will just add the new row at the end of the preallocated memory. Ideally, it would be smart enough to resize memory, if needed.
I may be grossly missing something, but I sincerly couldn't find a way to do that anywhere I looked.