title: Ktor - Handle large file operations without memory leak
body: <p>I am very new to backend development. Basically, I want to create a robust &amp; simple application that will accept a zip file URL in the params and then download the zip file from the URL and finally extract the zip and return the <code>bin</code> file inside it. Note: The zip file size can range from 5MB to 150MB. I have tried doing the described operation in the following manner.</p>
  <pre><code>package la.sample
  
  import io.ktor.application.Application
  import io.ktor.application.call
  import io.ktor.client.HttpClient
  import io.ktor.client.request.get
  import io.ktor.http.HttpStatusCode
  import io.ktor.response.respond
  import io.ktor.response.respondFile
  import io.ktor.routing.get
  import io.ktor.routing.routing
  import java.io.*
  
  
  fun Application.startServer() {
      routing {
          get(&quot;/get-bin&quot;) {
  
              //Gets the AWS Url from params
              val awsUrl = call.request.queryParameters.get(&quot;url&quot;) ?: &quot;Error&quot;
  
              // Download the zip file from the AWS URL
              val client = HttpClient()
              val bytes = client.get&lt;ByteArray&gt;(awsUrl)
  
              //Create a temp file on the server &amp; write the zip file bytes into it.
              val file = File(&quot;.&quot;, &quot;data.zip&quot;) 
              file.writeBytes(bytes) 
              
              //Call a method to unzip the file
              unzipAndReturnBinFile()?.let { 
                  call.respondFile(it) //respond with bin file
              } ?: kotlin.run{
                  call.respond(HttpStatusCode.InternalServerError)
              }
          }
      }
  }
  
  
  fun unzipAndReturnBinFile(): File? {
  
      var exitVal = 0
  
      //Command shell to unzip the file
      Runtime.getRuntime().exec(&quot;unzip bundle.zip -d data&quot;).let {//command shell to unzip the zip file
          exitVal += it.waitFor()
      }
  
      //Check if the command executed successfully 
      if (exitVal == 0) {
  
          var binFile: File? = null
  
          //check if the extracted files contain `bin`
          File(&quot;data&quot;).listFiles().forEach {
  
          if (it.name.contains(&quot;.bin&quot;)) {
              binFile = it
          }
      }
  
      //return bin or null otherwise
      return binFile
  } else {
      throw Exception(&quot;Command Shell Execution failed.&quot;)
  }
  }
  </code></pre>
  <p>The above codes work fine in local machine, irrespective of the Zip file size. But when it is deployed to AWS, the code breaks if the zip or the bin file is larger than 100 MB and gives a <code>java.lang.OutOfMemoryError</code> error. I will be very thankful if someone can suggest to me a proper way of handling large file operations in the backend with the ability to handle 100s of such concurrent calls. Thank you.</p>
  <p>Java heap size of my remote machine is around 1 GB.
  <a href="https://i.stack.imgur.com/5smn5.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/5smn5.png" alt="enter image description here" /></a></p>