title: Dealing with incremental server response in AJAX (in JavaScript)
body: <p>I have AJAX app which updates page based on server response.  The command that AJAX server response is based on takes long time to generate full response, but it sends partial information as soon as it is calculated. This partial response / partial info is send in "burst", and time and size of each burst is unpredictable.  CGI script (in Perl) that streams command output to web browser (to AJAX request) has autoflush turned on.</p>
  
  <p>The server response is based on output of external command. While 'time cmd > /dev/null" gives around 10.0 seconds on average, 'time cmd | head > /dev/null' gives less than 0.1 seconds (for example data). All data is result of single call to this external command. </p>
  
  <p>The situation looks like the following <em>(ASCII-art diagram follows)</em>:</p>
  
  <pre><code> client |     | server
  ---------     ---------
  
  request -\
            \
             \
              \
               \-&gt;
  
               /- response
              /      .
             /       .
            /  /-    .
         &lt;-/  /      . 
             /       .
            /  /-  [end]
         &lt;-/  /
             /
            /
         &lt;-/
  </code></pre>
  
  <p>I have a few questions about this problem.</p>
  
  <p><strong>Note:</strong> server side is done as CGI script in Perl, and I would prefer to see (also) solution without using JavaScript library / framework like jQuery.</p>
  
  <ul>
  <li><p>The output of command used by server side of AJAX app is line based. Each group of lines, beginning with one defined kind of line, and ending with other kind of line, consist of independend and unchangeable data. Should I just stream response from a command as 'text/plain' and do processing in JavaScript on client side, or should I pre-process data on server, and send whole chunks of data as JSON using 'application/json' mimetype?</p></li>
  <li><p>It might happen that large chunk of data send at once by server is followed soon by another chunk of data.  How to deal with situation when <code>onreadystatechange</code> handler is invoked while previous invocation didn't finished work?  Should I use global variable as semaphore, or pass state variable as handler parameter (well, use <code>xhr.onreadystatechange = function() { handleRequest(xhr, state) }</code>)?</p></li>
  <li><p>Should I use 'text/plain' or 'application/json', or perhaps 'multipart/x0mixed-replace' for this? Note: this app should work in (alomst) any browser.</p></li>
  <li><p>How to deal with web browser (JavaScript engines) which invoke onReadyStateChange only after receiving complete response (so I don't see <code>xhr.readyState == 3</code> i.e. partial response more than once)? Well, beside using some JavaScript framework.</p></li>
  <li><p>How to deal with incomplete responses (which in this situation mean incomplete lines).</p></li>
  <li><p>Should I send end of response marker, or rely on counter to check if we received all data, or can I simply rely on detecting <code>xhr.readyState == 4</code>?</p></li>
  </ul>
  
  <p>Even partial response would be helpful.</p>