Unpickling mid-stream (python)
Question: I am writing scripts to process (very large) files by repeatedly unpickling
objects until EOF. I would like to partition the file and have separate
processes (in the cloud) unpickle and process separate parts.

However my partitioner is not intelligent, it does not know about the
boundaries between pickled objects in the file (since those boundaries depend
on the object types being pickled, etc.).

Is there a way to scan a file for a "start pickled object" sentinel? The naive
way would be to attempt unpickling at successive byte offsets until an object
is successfully pickled, but that yields unexpected errors. It seems that for
certain combinations of input, the unpickler falls out of sync and returns
nothing for the rest of the file (see code below).

    
    
    import cPickle
    import os
    
    def stream_unpickle(file_obj):
        while True:
            start_pos = file_obj.tell()
            try:
                yield cPickle.load(file_obj)
            except (EOFError, KeyboardInterrupt):
                break
            except (cPickle.UnpicklingError, ValueError, KeyError, TypeError, ImportError):
                file_obj.seek(start_pos+1, os.SEEK_SET)
    
    if __name__ == '__main__':
        import random
        from StringIO import StringIO
    
        # create some data
        sio = StringIO()
        [cPickle.dump(random.random(), sio, cPickle.HIGHEST_PROTOCOL) for _ in xrange(1000)]
        sio.flush()
    
        # read from subsequent offsets and find discontinuous jumps in object count
        size = sio.tell()
        last_count = None
        for step in xrange(size):
            sio.seek(step, os.SEEK_SET)
            count = sum(1 for _ in stream_unpickle(file_obj))
            if last_count is None or count == last_count - 1:
                last_count = count
            elif count != last_count:
                # if successful, these should never print (but they do...)
                print '%d elements read from byte %d' % (count, step)
                print '(%d elements read from byte %d)' % (last_count, step-1)
                last_count = count
    


Answer: The _[pickletools](http://docs.python.org/library/pickletools.html#module-
pickletools)_ module has a _dis_ function that shows the opcodes. It shows
that there is a STOP opcode that you may be scan for:

    
    
    >>> import pickle, pickletools, StringIO
    >>> s = StringIO.StringIO()
    >>> pickle.dump('abc', s)
    >>> p = s.getvalue()
    >>> pickletools.dis(p)
        0: S    STRING     'abc'
        7: p    PUT        0
       10: .    STOP
    highest protocol among opcodes = 0
    

Note, using the STOP opcode is a bit tricky because the codes are of variable
length, but it may serve as a useful hint about where the cutoffs are.

If you control the pickling step on the other end, then you can improve the
situation by adding your own unambiguous alternative separator:

    
    
    >>> sep = '\xDE\xAD\xBE\xEF'
    >>> s = StringIO.StringIO()
    >>> pickle.dump('abc', s)
    >>> s.write(sep)
    >>> pickle.dump([10, 20], s)
    >>> s.write(sep)
    >>> pickle.dump('def', s)
    >>> s.write(sep)
    >>> pickle.dump([30, 40], s)
    >>> p = s.getvalue()
    

Before unpacking, split into separate pickles using the known separator:

    
    
    >>> for pick in p.split(sep):
            print pickle.loads(pick)
    
    abc
    [10, 20]
    def
    [30, 40]
    

