title: Handling logging over data stream
body: <p>I have an EC2 instance that has a datastream that uses an event emitter to process data. E.g. </p>
  
  <pre><code>stream.on('new event', function doSomething(event){ do more stuff...})
  </code></pre>
  
  <p>This datastream has potentially tens of thousands of events per second, and I would like to log the processing of these events in an efficient manner. In other words, I'm reluctant to send out a log entry each time a new event comes in. </p>
  
  <p>Hence, I figured I'd send logs in batch. E.g.</p>
  
  <pre><code>let logArray = [];
  function sendToLogs(logs) {\** send stuff *\}
  
  stream.on('new event', function doSomething(event){ 
    \\do some stuff
  
    logArray.push({newLog: event})
    if (logArray.length &gt;= 500) {
       sendToLogs(logArray)
       logArray = [];
    }
  })
  </code></pre>
  
  <p>However, I'm afraid that with so many events coming in simultaneously, that the above code could lead to erratic behavior. I've seen this in local logging: the length of this array jumps around pretty dramatically and can have the same value for different events simultaneously. </p>
  
  <p>Furthermore, using cloudwatch logs requires me to pass 'sequenceTokens' between different calls to the logging function. If two events trigger the logging condition at the same time, things could get weird. (This problem would exist even if I logged each event separately.)</p>
  
  <p>How should I handle logging over this kind of data stream?</p>