title: How to receive and assemble byte arrays with variable lengths in Python sockets?
body: <p>I am trying to send large <strong>byte arrays</strong> of a Protobuf class from a Java client to the Python server. However, they have a <strong>variable</strong> length, because sometimes I send the bytes of an object from <code>ClassA</code> and sometimes from <code>ClassB</code>.</p>
  
  <p>I have a Python socket server with the following code inside the function that listens to the socket:</p>
  
  <pre class="lang-py prettyprint-override"><code>byte_array = bytearray()
  
  # receive the data in small chunks and print it
  while True:
      data = connection.recv(64)
      if data:
          # output received data
          logger.debug("Data: %s" % data)
          byte_array.extend(data)
  
      else:
          # no more data -- quit the loop
          logger.debug("no more data.")
          break
  
  logger.info("Generating response...")
  connection.send(generate_response(byte_array))
  logger.info("Sent response.")
  </code></pre>
  
  <p>I am assembling the large byte array that I receive by putting together the 64 bytes as I get them.</p>
  
  <p>However, when the byte array is fully transmitted and there is nothing left to send, the server hangs on the <code>connection.recv</code> line. </p>
  
  <p>I read that this is because <code>recv</code> blocks until either it receives something or the connection is closed. However, I do not want to close the connection because I want to send my response back to the client after processing the whole byte array.</p>
  
  <p><strong>I want to know when the byte array I am receiving has been fully transmitted,</strong> so that I can avoid this blocking.</p>
  
  <p>I can think of three options:</p>
  
  <ul>
  <li>Set a predefined "end" byte, delimiting the end of the byte array.</li>
  <li>Send the size of the byte array beforehand and then instead of <code>while True</code> I have a <code>while bytes_read &lt; expected_bytes</code> cycle.</li>
  <li>Set a timeout on the connection and I assume that when a timeout occurs it means everything has already been sent.</li>
  </ul>
  
  <p>I am inclined for the first option, however I do not know what character I should use to end the byte array nor how to read it in my Python code.</p>
  
  <p>Any suggestions?</p>
  
  <p>Thanks.</p>