Persistence of a large number of objects
Question: I have some code that I am working on that scrapes some data from a website,
and then extracts certain key information from that website and stores it in
an object. I create a couple hundred of these objects each day, each from
unique url's. This is working quite well, however, I'm inexperienced in what
options are available to me in Python for persistence and what would be best
suited for my needs.

Currently I am using pickle. To do so, I am keeping all of these webpage
objects and appending them in a list as new ones are created, then saving that
list to a pickle (then reloading it whenever the list is to be updated).
However, as i'm in the order of some GB of data, i'm finding pickle to be
somewhat slow. It's not unworkable, but I'm wondering if there is a more well
suited alternative. I don't really want to break apart the structure of my
objects and store it in a sql type database, as its important for me to keep
the methods and the data as a single object.

Shelve is one option I've been looking into, as my impression is then that I
wouldn't have to unpickle and pickle all the old entries (just the most recent
day that needs to be updated), but am unsure if this is how shelve works, and
how fast it is.

So to avoid rambling on, my question is: what is the preferred persistence
method for storing a large number of objects (all of the same type), to keep
read/write speed up as the collection grows?


Answer: Martijn's suggestion could be one of the alternatives.

You may consider to store the pickle objects directly in a sqlite database
which still can manage from the python standard library.

Use a StringIO object to convert between the database column and python
object.

You didn't mention the size of each object you are pickling now. I guess it
should stay well within sqlite's limit.

