title: Parsing data from a network stream?
body: <p>Recently I started working with sockets. I realized that when reading from a network stream, you can not know how much data is coming in. So either you know in advance <strong>how many bytes</strong> have to be recieved or you know <strong>which bytes</strong>.</p>
  <p>Since I am currently trying to implement a C# <a href="https://www.rfc-editor.org/rfc/rfc6455" rel="nofollow noreferrer">WebSocket</a> server I need to process HTTP requests. A HTTP request can have arbitrary length, so knowing in advance <em>how many bytes</em> is out of the question. But a HTTP request always has a <a href="https://www.rfc-editor.org/rfc/rfc2616" rel="nofollow noreferrer">certain format</a>. It starts with the request-line, followed by zero or more headers, etc. So with all this information it should be simple, right?</p>
  <p>Nope.</p>
  <p>One approach I came up with was reading all data until a specific sequence of bytes was recognized. The <a href="http://msdn.microsoft.com/en-us/library/system.io.streamreader.aspx" rel="nofollow noreferrer">StreamReader class</a> has the <code>ReadLine</code> method which, I believe, works like this. For HTTP a reasonable delimiter would be the <em>empty line</em> separating the message head from the body.</p>
  <p>The obvious problem here is the requirement of a (preferrably short) termination sequence, like a line break. Even the HTTP specification suggests that these two adjacent CRLFs are not a good choice, since they could also occur at the beginning of the message. And after all, two CRLFs are not a simple delimiter anyways.</p>
  <p>So expanding the method to arbitrary type-3 grammars, I concluded the best choice for parsing the data is a finite state machine. I can feed the data to the machine byte after byte, just as I am reading it from the network stream. And as soon as the machine accepts the input I can stop reading data. Also, the FSM could immediately capture the significant tokens.</p>
  <p>But is this really the best solution? Reading byte after byte and validating it with a custom parser seems tedious and expensive. And the FSM would be either slow or quite ugly. So...</p>
  <p><strong>How do you process data from a network stream when the form is known but not the size?</strong></p>
  <p><strong>How can classes like the <a href="http://msdn.microsoft.com/en-us/library/system.net.httplistener.aspx" rel="nofollow noreferrer">HttpListener</a> parse the messages and be fast at it too?</strong></p>
  <p>Did I miss something here? How would this usually be done?</p>