python example for reading multiple protobuf messages from a stream
Question: I'm working with data from spinn3r, which consists of multiple different
protobuf messages serialized into a byte stream:

<http://code.google.com/p/spinn3r-client/wiki/Protostream>

"A protostream is a stream of protocol buffer messages, encoded on the wire as
length prefixed varints according to the Google protocol buffer specification.
The stream has three parts: a header, the payload, and a tail marker."

This seems like a pretty standard use case for protobufs. In fact, protobuf
core distribution provides CodedInputStream for both C++ and Java. But, it
appears that protobuf does not provide such a tool for python -- the
'internal' tools are not setup for this kind of external use:

<https://groups.google.com/forum/?fromgroups#!topic/protobuf/xgmUqXVsK-o>

So... before I go and cobble together a python varint parser and tools for
parsing a stream of different message types: does anyone know of any tools for
this?

Why is it missing from protobuf? (Or am I just failing to find it?)

This seems like a big gap for protobuf, especially when compared to thrift's
equivalent tools for both 'transport' and 'protocol'. Am I viewing that
correctly?


Answer: I looks like the code in the other answer is potentially lifted from
[here](https://code.google.com/r/avenka-
co1/source/browse/src/python/varint.py?spec=svn2b97a552eb2c39cfe62b62dbc303b6e86c47ec4b&name=series34&r=2b97a552eb2c39cfe62b62dbc303b6e86c47ec4b).
Check the licence before using this file but I managed to get it to read
`varint32`s using code such as this:

    
    
    import sys
    import myprotocol_pb2 as proto
    import varint # (this is the varint.py file)
    
    data = open("filename.bin", "rb").read() # read file as string
    decoder = varint.decodeVarint32          # get a varint32 decoder
                                             # others are available in varint.py
    
    next_pos, pos = 0, 0
    while 1:
        msg = proto.Msg()                    # your message type
        next_pos, pos = decoder(data, pos)
        msg.ParseFromString(data[pos:pos + next_pos])
    
        if msg.type == proto.Msg.END:        # example check for end
            break
    
        # use parsed message
    
        pos += next_pos
    print "done!"
    

This is very simple code designed to load messages of a single type delimited
by `varint32`s which describe the next message's size. Watch for the infinite
loop and add error handling.

