title: How do I read binary C++ protobuf data using Python protobuf?
body: <p>The Python version of Google protobuf gives us only:</p>
  
  <pre><code>SerializeAsString()
  </code></pre>
  
  <p>Where as the C++ version gives us both:</p>
  
  <pre><code>SerializeToArray(...)
  SerializeAsString()
  </code></pre>
  
  <p>We're writing to our C++ file in binary format, and we'd like to keep it this way. That said, is there a way of reading the binary data into Python and parsing it as if it were a string?</p>
  
  <p><s>Is this the correct way of doing it?</p>
  
  <pre><code>binary = get_binary_data()
  binary_size = get_binary_size()
  
  string = None
  for i in range(len(binary_size)):
     string += i
  
  message = new MyMessage()
  message.ParseFromString(string)
  </code></pre>
  
  <p></s></p>
  
  <h2>Update:</h2>
  
  <p>Here's a new example, and a problem:</p>
  
  <pre><code>message_length = 512
  
  file = open('foobars.bin', 'rb')
  
  eof = False
  while not eof:
  
      data = file.read(message_length)
      eof = not data
  
      if not eof:
          foo_bar = FooBar()
          foo_bar.ParseFromString(data)
  </code></pre>
  
  <p>When we get to the <code>foo_bar.ParseFromString(data)</code> line, I get this error:</p>
  
  <pre><code>Exception Type: DecodeError
  Exception Value: Too many bytes when decoding varint.
  </code></pre>
  
  <h2>Update 2:</h2>
  
  <p>It turns out, that the padding on the binary data was throwing protobuf off; too many bytes were being sent in, as the message suggests (in this case it was referring to the padding). </p>
  
  <p>This padding comes from using the C++ protobuf function, <code>SerializeToArray</code> on a fixed-length buffer. To eliminate this, I have used this temproary code:</p>
  
  <pre><code>message_length = 512
  
  file = open('foobars.bin', 'rb')
  
  eof = False
  while not eof:
  
      data = file.read(message_length)
      eof = not data
  
      string = ''
      for i in range(0, len(data)):
          byte = data[i]
          if byte != '\xcc': # yuck!
              string += data[i]
  
      if not eof:
          foo_bar = FooBar()
          foo_bar.ParseFromString(string)
  </code></pre>
  
  <p>There is a design flaw here I think. I will re-implement my C++ code so that it writes variable length arrays to the binary file. As advised by the protobuf documentation, I will prefix each message with it's binary size so that I know how much to read when I'm opening the file with Python.</p>