title: AWS CPP TransferManager vs GetObjectRequest Stream to File fstream OOM
body: <p>I am using the AWS CPP SDK ( <a href="https://github.com/aws/aws-iot-device-sdk-cpp" rel="nofollow noreferrer">https://github.com/aws/aws-iot-device-sdk-cpp</a> ) to download a file from S3 on a small Linux system ( only 32 MB RAM ). I was using the GetObjectRequest class as shown below. It worked great and downloaded the file to the FStream on my system so that it did not consume too much RAM.</p>
  
  <p>Now, I want to convert the download code to the TransferManager method in order to get progress callbacks. I've rewritten that part of the code and it is shown below as well. It starts out fine, prints the percentage downloaded, but when it reached ~14 MB RAM ( roughly the amount available in Linux at the time of download ) it is killed by the kernel for using too much RAM. </p>
  
  <p>I created a file stream just like I did for the GetObjectRequest. What am I doing wrong? How can I fix this? Thanks.</p>
  
  <p>Old way that did not use all the RAM:</p>
  
  <pre><code>    // Old way
      GetObjectRequest getObjectRequest;
      getObjectRequest.SetBucket(bucket.c_str());
      getObjectRequest.SetKey(keyName.c_str());
      getObjectRequest.SetResponseStreamFactory([&amp;destination](){
       return Aws::New&lt;Aws::FStream&gt;(
       "s3file", destination, std::ios_base::out); });
  
      GetObjectOutcome getObjectOutcome = SessionClient-&gt;GetObject(getObjectRequest);
      if(getObjectOutcome.IsSuccess())
      {
          std::cout &lt;&lt; "&lt;AWS DOWNLOAD&gt; Get FW success!" &lt;&lt; std::endl;
      }
      else
      {
          std::cout &lt;&lt; "&lt;AWS DOWNLOAD&gt; Get FW failed: " &lt;&lt; getObjectOutcome.GetError().GetMessage() &lt;&lt; std::endl;
          exit(1);
      }
  </code></pre>
  
  <p>New way that eventually uses too much RAM and is killed by the kernel:</p>
  
  <pre><code>// New way
  Aws::Transfer::TransferManagerConfiguration transferConfig;
  transferConfig.s3Client = SessionClient;
  
  std::shared_ptr&lt;Aws::Transfer::TransferHandle&gt; requestPtr(nullptr);
  
  transferConfig.downloadProgressCallback =
          [](const Aws::Transfer::TransferManager*, const Aws::Transfer::TransferHandle&amp; handle)
  {
      std::cout &lt;&lt; "\r" &lt;&lt; "&lt;AWS DOWNLOAD&gt; Download Progress: " &lt;&lt; static_cast&lt;int&gt;(handle.GetBytesTransferred() * 100.0 / handle.GetBytesTotalSize()) &lt;&lt; " Percent " &lt;&lt; handle.GetBytesTransferred() &lt;&lt; " bytes\n";
  };
  
  Aws::Transfer::TransferManager transferManager(transferConfig);
  
  requestPtr = transferManager.DownloadFile(bucket.c_str(), keyName.c_str(), [&amp;destination](){
  
      Aws::FStream *stream = Aws::New&lt;Aws::FStream&gt;("s3file", destination, std::ios_base::out);
      stream-&gt;rdbuf()-&gt;pubsetbuf(NULL, 0);
  
      return stream; });
  
  requestPtr-&gt;WaitUntilFinished();
  
  size_t retries = 0;
  //just make sure we don't fail because a download part failed. (e.g. network problems or interuptions)
  while (requestPtr-&gt;GetStatus() == Aws::Transfer::TransferStatus::FAILED &amp;&amp; retries++ &lt; 5)
  {
      std::cout &lt;&lt; "&lt;AWS DOWNLOAD&gt; FW Download trying download again!" &lt;&lt; std::endl;
      transferManager.RetryDownload(requestPtr);
      requestPtr-&gt;WaitUntilFinished();
  }
  
  // Check status
  if ( requestPtr-&gt;GetStatus() == Aws::Transfer::TransferStatus::COMPLETED ) {
      if ( requestPtr-&gt;GetBytesTotalSize() == requestPtr-&gt;GetBytesTransferred() ) {
          std::cout &lt;&lt; "&lt;AWS DOWNLOAD&gt; Get FW success!" &lt;&lt; std::endl;
          exit(0);
      }
      else {
          std::cout &lt;&lt; "&lt;AWS DOWNLOAD&gt; Get FW failed - Bytes downloaded did not equal requested number of bytes: " &lt;&lt; requestPtr-&gt;GetBytesTotalSize() &lt;&lt; requestPtr-&gt;GetBytesTransferred() &lt;&lt; std::endl;
          exit(1);
      }
  }
  else {
      std::cout &lt;&lt; "&lt;AWS DOWNLOAD&gt; Get FW failed - download was never completed even after retries" &lt;&lt; std::endl;
      exit(1);
  }
  </code></pre>