title: Nodejs Cluster Architecture reading from single REDIS instance
body: <p>I'm using Nodejs <code>cluster</code> module to have multiple workers running.
  I created a basic Architecture where there will be a single MASTER process which is basically an express server handling multiple requests and the main task of MASTER would be writing incoming data from requests into a REDIS instance. Other workers(numOfCPUs - 1) will be non-master i.e. they won't be handling any request as they are just the consumers. I have two features namely ABC and DEF. I distributed the non-master workers evenly across features via assigning them type.</p>
  
  <p>For eg: on a 8-core machine:</p>
  
  <p>1 will be MASTER instance handling request via express server</p>
  
  <p>Remaining (8 - 1 = 7) will be distributed evenly. 4 to feature:ABD and 3 to fetaure:DEF.</p>
  
  <p>non-master workers are basically consumers i.e. they read from REDIS in which only MASTER worker can write data.</p>
  
  <p>Here's the code for the same:</p>
  
  <pre><code>if (cluster.isMaster) {
    // Fork workers.
    for (let i = 0; i &lt; numCPUs - 1; i++) {
      ClusteringUtil.forkNewClusterWithAutoTypeBalancing();
    }
  
    cluster.on('exit', function(worker) {
      console.log(`Worker ${worker.process.pid}::type(${worker.type}) died`);
      ClusteringUtil.removeWorkerFromList(worker.type);
      ClusteringUtil.forkNewClusterWithAutoTypeBalancing();
    });
  
    // Start consuming on server-start
    ABCConsumer.start();
    DEFConsumer.start();
  
    console.log(`Master running with process-id: ${process.pid}`);
  } else {
    console.log('CLUSTER  type', cluster.worker.process.env.type, 'running on', process.pid);
    if (
      cluster.worker.process.env &amp;&amp;
      cluster.worker.process.env.type &amp;&amp;
      cluster.worker.process.env.type === ServerTypeEnum.EXPRESS
    ) {
      // worker for handling requests
      app.use(express.json());
      ...
    }
  {
  
  </code></pre>
  
  <p>Everything works fine except consumers reading from REDIS.
  Since there are multiple consumers of a particular feature, each one reads the same message and start processing individually, which is what I don't want. If there are 4 consumers, 1 is marked as busy and can not consumer until free, 3 are available. Once the message for that particular feature is written in REDIS by MASTER, the problem is all 3 available consumers of that feature start consuming. This means that the for a single message, the job is done based on number of available consumers.</p>
  
  <pre><code>const stringifedData = JSON.stringify(req.body);
    const key = uuidv1();
  
    const asyncHsetRes = await asyncHset(type, key, stringifedData);
  
    if (asyncHsetRes) {
      await asyncRpush(FeatureKeyEnum.REDIS.ABC_MESSAGE_QUEUE, key);
      res.send({ status: 'success', message: 'Added to processing queue' });
    } else {
      res.send({ error: 'failure', message: 'Something went wrong in adding to queue' });
    }
  </code></pre>
  
  <p>Consumer simply accepts messages and stop when it is busy</p>
  
  <pre><code>module.exports.startHeartbeat = startHeartbeat = async function(config = {}) {
    if (!config || !config.type || !config.listKey) {
      return;
    }
  
    heartbeatIntervalObj[config.type] = setInterval(async () =&gt; {
      await asyncLindex(config.listKey, -1).then(async res =&gt; {
        if (res) {
          await getFreeWorkerAndDoJob(res, config);
          stopHeartbeat(config);
        }
      });
    }, HEARTBEAT_INTERVAL);
  };
  </code></pre>
  
  <p>Ideally, a message should be read by only one consumer of that particular feature. After consuming, it is marked as busy so it won't consume further until free(I have handled this). Next message could only be processed by only one consumer out of other available consumers.</p>
  
  <p>Please help me in tacking this problem. Again, I want one message to be read by only one free consumer and rest free consumers should wait for new message.</p>
  
  <p>Thanks</p>