title: Storm executors stop processing in a few hours
body: <p>Storm experts,</p>
  
  <p>We are using storm 0.8.2 and have a topology with 1 spout, which emits tuples to Bolt 1, which does some processing, web calls &amp; filtering &amp; emits to Bolt 2. Bolt 1 to Bolt 2 have a reduction of 20% and the new Tuples emitted are only 80% of total Tuples received.
  Bolt 2  does further processing &amp; db calls and emits tuples to Bolt 3.  Bolt 2 discards almost 50%  of unnecessary messages and creates new Tuples to emit only for 50%,
  Bolt 3 makes a lot of web calls, running in threads (10 threads spawned per executor) and emits new Tuples to Bolt 4 for almost 90% of incoming messages.
  Bolt 4 does final processing on received messages and sends to a database.</p>
  
  <p>We are running 1 spout, 50 executors of Bolt 1, 8 executors of Bolt 2, 120 executors of Bolt 3 and 100 executors of Bolt 4. They are running on 8 workers on an 8 core machine in AWS.</p>
  
  <p>The problem we are facing is in a few hours, one of the workers (any of the 7 at random, no observable pattern) stops emitting. We added some log messages to check if execute is being called in each executor at this time, and as expected, the execute is not called as the message is not in log file during that time.</p>
  
  <p>When this worker is the one with spout running on it, the entire topology has zero emissions all across.</p>
  
  <p>surprisingly, the nimbus or supervisor does not restart any of the executors/workers as the heartbeat is there. The log file shows no exceptions. Also, its not for a single executor but for the entire worker. Killing the worker via sudo kill pid restarts the message flow when the worker is alive again and restarts the executors on that worker.</p>
  
  <p>Is there a way to monitor pending messages in Transfer Queues/Executor Queue/ ZeroMQ? Perhaps that will help us debug better. 
  Has anyone faced such a problem earlier? Does it have something to do with the number of tuples getting reduced at each Bolt? </p>