title: Understanding the Storm Architecture
body: <p>I have been trying to understand the storm architecture, but I am not sure if I got this right. I'll try to explain as exactly as possible what I believe to be the case. Please explain what - if - I got wrong and what is right.</p>
  
  <h2>Preliminary thoughts: <em>workers</em></h2>
  
  <p><a href="http://storm.apache.org/releases/2.0.0-SNAPSHOT/Understanding-the-parallelism-of-a-Storm-topology.html" rel="nofollow">http://storm.apache.org/releases/2.0.0-SNAPSHOT/Understanding-the-parallelism-of-a-Storm-topology.html</a> suggests that a Worker is a process as does <a href="http://storm.apache.org/releases/2.0.0-SNAPSHOT/Concepts.html" rel="nofollow">http://storm.apache.org/releases/2.0.0-SNAPSHOT/Concepts.html</a> with "worker processes. Each worker process is a physical JVM", however <a href="http://storm.apache.org/releases/1.0.1/Setting-up-a-Storm-cluster.html" rel="nofollow">http://storm.apache.org/releases/1.0.1/Setting-up-a-Storm-cluster.html</a> states that a worker is a node with "Nimbus and worker machines". The website <a href="http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/" rel="nofollow">http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/</a> mentions "master node" and "worker nodes". So what: Is the worker a process or a physical node (or is a node a process)? Thus I think that there are two things: <em>Worker Nodes</em> and <em>Worker Processes</em>.</p>
  
  <h2>What I believe to be true</h2>
  
  <h3>Entities in play</h3>
  
  <ul>
  <li><em>Master Node</em> = Management Server</li>
  <li><em>Worker Node</em> = <em>Slave Node</em></li>
  <li><em>Nimbus</em> JVM process, running on <em>Master Node</em></li>
  <li><em>ZooKeeper</em> JVM processes, running on <em>ZooKeeper Nodes</em></li>
  <li><em>Supervisor</em> JVM process, running on <em>Worker Nodes</em></li>
  <li><em>Worker Process</em> (JVM), running on <em>Worker Nodes</em></li>
  <li><em>Executor</em> thread, run by <em>Worker Process</em></li>
  <li><em>Task</em> (instances of <em>Bolts</em> and <em>Spouts</em>), executed by <em>Executor</em></li>
  </ul>
  
  <h3>How things work</h3>
  
  <p>The <em>Nimbus</em> is a JVM process, running on the physical <em>Master Node</em>, that receives my program (<em>Storm topology</em>) takes the <em>Bolts and Spouts</em> and generates <em>tasks</em> from them. If a <em>Bolt</em> is supposed to be parallelized three times, the <em>Nimbus</em> generates three <em>tasks</em> for it. The <em>Nimbus</em> asks the <em>ZooKeeper</em> JVM processes about configurations of the cluster, e.g. where to run those <em>tasks</em> and the <em>ZooKeeper</em> JVM processes tells the <em>Nimbus</em>. In order to do this the <em>ZooKeeper</em> communicates with the <em>Supervisors</em> (what they are comes later). The <em>Nimbus</em> distributes the tasks then to the <em>Workers Nodes</em>, which are physical nodes. <em>Worker Nodes</em> are managed by <em>Supervisors</em>, which are JVM processes - exactly one <em>Supervisor</em> for one <em>Worker Node</em>. <em>Supervisors</em> manage (start, stop, etc.) the <em>Worker Processes</em>, which are JVM processes that run on the <em>Worker Nodes</em>. Each <em>Worker Node</em> can have multiple <em>Worker Processes</em> running. <em>Worker Processes</em> are JVM processes. They run one or multiple threads called <em>Executors</em>. Each <em>Executor Thread</em> runs one or multiple <em>Tasks</em>, meaning one or multiple instances of a <em>Bolt</em> or <em>Spout</em>, <a href="http://storm.apache.org/releases/2.0.0-SNAPSHOT/Understanding-the-parallelism-of-a-Storm-topology.html" rel="nofollow">but those have to be of the same <em>Bolt</em></a>. </p>
  
  <p>If this is all true, it begs the questions:</p>
  
  <ul>
  <li>What is the point of having multiple <em>Worker Processes</em> run on one <em>Worker Node</em> - after all a process can use multiple processor cores, right?</li>
  <li>What is the point of having multiple <em>Tasks</em> run by one <em>Executor</em> thread if they have to be of the same <em>Bolt</em>/<em>Spout</em>? A thread is only run on one processor core, so the multiple <em>Bolt</em>/<em>Spout</em> have to run after each other and can not be parallelized. Running indefinitely in the Storm topology - what would be the point having two instances of the same <em>Bolt</em>/<em>Spout</em> in a <em>Exectuor</em> thread?</li>
  </ul>
  
  <hr>
  
  <p><strong>Edit:</strong> Good additional resource: <a href="http://www.tutorialspoint.com/apache_storm/apache_storm_cluster_architecture.htm" rel="nofollow">http://www.tutorialspoint.com/apache_storm/apache_storm_cluster_architecture.htm</a></p>