title: Memory buildup in Node streams
body: <p>tl;dr: if you have multiple steps in a stream process (A -> B -> C), and the start of the stream (A -> B) is able to pull in data much faster than the output stream is able to send it off (B -> C), does that cause a buildup in memory consumption?</p>
  
  <p>I'm still a beginner at Node streams and want to check in about my understanding...</p>
  
  <p>I'm building a tool that pulls down photos from S3, streams them into a zip archive, and streams that into an S3 upload. I was using streams because I thought it would be possible to have data move quickly and not take up too much memory.</p>
  
  <p>However I'm finding that the memory does steadily grow as the script runs and pulls down more and more photos (we're talking on the order of 1,000s of photos).</p>
  
  <p>My question is whether this is due to an issue of backpressure in the system, e.g. that the upload stream is not sending off data as quickly as the download streams are piping into the zip stream.</p>
  
  <p>Here's the code with a bunch of error handling and logging omitted for clarity.</p>
  
  <pre><code>const AWS = require('aws-sdk')
  const s3Zip = require('s3-zip')
  
  exports.handler = function (event, context) {
    const { region, bucket, folder, files, zipFileName } = event
    /**
     * s3Zip.archive is a library call that reads multiple S3 objects using #getObject().createReadStream() and pipes
     * them into a zip using the `archiver` library
     */
    const body = s3Zip.archive({ region, bucket }, folder, files)
    /**
     * s3.upload takes a stream Body and uploads it to s3
     * https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#upload-property
     */
    const upload = s3.upload({ Bucket: bucket, Key: zipFileName, Body: body })
    upload.send((err, result) =&gt; { /* error handling or trigger a success */ })
  }
  </code></pre>