Why doesn't appending binary pickles work?
Question: I know this isn't exactly how the pickle module was intended to be used, but I
would have thought this would work. I'm using Python 3.1.2

Here's the background code:

    
    
    import pickle
    
    FILEPATH='/tmp/tempfile'
    
    class HistoryFile():
        """
        Persistent store of a history file  
        Each line should be a separate Python object
        Usually, pickle is used to make a file for each object,
            but here, I'm trying to use the append mode of writing a file to store a sequence
        """
    
        def validate(self, obj):
            """
            Returns whether or not obj is the right Pythonic object
            """
            return True
    
        def add(self, obj):
            if self.validate(obj):
                with open(FILEPATH, mode='ba') as f:    # appending, not writing
                    f.write(pickle.dumps(obj))
            else:
                raise "Did not validate"
    
        def unpack(self):
            """
            Go through each line in the file and put each python object
            into a list, which is returned
            """
            lst = []
            with open(FILEPATH, mode='br') as f:
                # problem must be here, does it not step through the file?
                for l in f:
                    lst.append(pickle.loads(l))
            return lst
    

Now, when I run it, it only prints out the first object that is passed to the
class.

    
    
    if __name__ == '__main__':
    
        L = HistoryFile()
        L.add('a')
        L.add('dfsdfs')
        L.add(['dfdkfjdf', 'errree', 'cvcvcxvx'])
    
        print(L.unpack())       # only prints the first item, 'a'!
    

Is this because it's seeing an early EOF? Maybe appending is intended only for
ascii? (in which case, why is it letting me do mode='ba'?) Is there a much
simpler duh way to do this?


Answer: Why would you think appending binary pickles would produce a single pickle?!
Pickling lets you put (and get back) several items one after the other, so
obviously it must be a "self-terminating" serialization format. Forget lines
and just get them back! For example:

    
    
    >>> import pickle
    >>> import cStringIO
    >>> s = cStringIO.StringIO()
    >>> pickle.dump(23, s)
    >>> pickle.dump(45, s)
    >>> s.seek(0)
    >>> pickle.load(s)
    23
    >>> pickle.load(s)
    45
    >>> pickle.load(s)
    Traceback (most recent call last):
       ...
    EOFError
    >>> 
    

just catch the `EOFError` to tell you when you're done unpickling.

