Most efficient datatype for iteratively adding to?
In your question you say when the data collection is finished (possibly months from now). It is enormous amount of time in comparison with efficiency of python or pandas or any other programming tool I can imagine. I just created 100k random dictionaries of length 18 containing floats, saved them into text file (csv format) and loaded with pandas into dataframe. Took 2 seconds to save and 0.5 seconds to load. So just add every new record to the file and periodically create backups of your choice.
I have a web scraper which iteratively retrieves data from web pages, and I would like to add the attributes pulled to a pandas dataframe (eventually) for running simple statistics and analysis. The current script returns a dictionary every time a new page is scraped.
I understand adding a new row or column to an existing pandas dataframe is slow, so my thought was to add the dictionary's as they are retrieved to a csv, and then convert this csv all at once to a dataframe when the data collection is finished (possibly months from now). I will be dealing with up to 100,000 dicts, with 18 key value pairs. 
Is there a more efficient method or datatype to use in this scenario?