title: Find the source of waves of latency from Redis using Node.js
body: <p>I am investigating some latency issues on my server, and I've narrowed it down but not enough to solve it. I'm hoping someone with more experience with Redis or Node.js can help.</p>
  <p>Within a function that is called a few thousand times per minute, scaling up and down with web traffic, I send a <code>GET</code> request to my redis client to check if a process is complete. I've noticed increased latency for my web requests, and it appears as though the redis <code>GET</code> command is taking up the bulk of my server time. Which surprised me, as I always thought redis was wicked fast all the time. And if I look at Redis's &quot;time spent&quot; info, it says everything is under 700 microseconds.</p>
  <p>That didn't jive with what I was seeing from my transaction monitoring setup, so I added some logging to my code:</p>
  <pre><code>const start = Date.now();
  client.get(`poll:${submittedId}`, (err, res) =&gt; {
      console.log(`${Date.now() - start}`);
      //other stuff
  })
  </code></pre>
  <p>Now my logs print the number of milliseconds spend on each redis <code>GET</code>. I watch that for a while, and see a surprising pattern.</p>
  <p>Most of the time, there are lots of <code>1</code>s and an occasional number in the 10s or sometimes 100s. Then, periodically, all the gets across the server slow down, <strong>reaching up to several seconds for each get to complete</strong>. Then after a while the numbers curve back down and things are running smoothly again.</p>
  <p><strong>What could be causing this sort of behaviour?</strong></p>
  <p>Things I've tried to investigate:</p>
  <ol>
  <li>Like I mentioned, I've combed through redis's performance data, as presented on Heroku's redis dashboard, and it doesn't have any complaints or latency spikes.</li>
  <li>I confirmed that all these requests are coming from a small number of connections, because I read that opening and closing too many can cause issues.</li>
  <li>I looked into connection pooling, thinking maybe the transactions are being queued and causing a backlog, but the internet seems to say this isn't necessary for Redis and Node.</li>
  </ol>
  <p>Really appreciate anyone taking the time to advise on this!</p>