title: How to log/instrument in performance critical web service?
body: <p>I'm working on a server push service, kind of like pusher/pubnub. The most critical part that handles the client polling is currently using Node.js and Redis, which is working just fine, except for one thing. I don't really have a good idea about what's going on in the app.</p>
  
  <p>The whole idea is based around long polling, which means there is A LOT of requests going back and forth, and a lot of checking in redis and seeing if there's something new. The problem is that I don't really know how to monitor such a thing.</p>
  
  <p>Let's say that there are 10 000 users online on average on a single site. At a polling interval of 5 seconds this would result in at least 2000 log entries every second. How do I manage that many logs? Should I use something like logstash to collect them to have at least some idea of what's going on in the app?</p>
  
  <p>Would it be a good idea to have all instrumentation turned off, and only enable it with something like <code>kill -s USR2</code>?</p>
  
  <p>I thought about using Redis <code>monitor</code> command to gather some data, but just running it slows down redis by 50%, not to mention the overhead of analyzing the incoming data.</p>
  
  <p>How do people generally handle this? Is there any good book or something about building high-load &amp; high-availability applications?</p>