title: Handling big files with Google Cloud Storage API
body: <p>What I need to achieve is to concatenate a list of files into a single file, using the <code>cloudstorage</code> library. This needs to happen inside a mapreduce shard, which has a 512MB upper limit on memory, but the concatenated file could be larger than 512MB. </p>
  
  <p>The following code segment breaks when file size hit the memory limit.</p>
  
  <pre><code>list_of_files = [...]
  with cloudstorage.open(filename...) as file_handler:
      for a in list_of_files:
          with cloudstorage.open(a) as f:
              file_handler.write(f.read())
  </code></pre>
  
  <p>Is there a way to walk around this issue? Maybe open or append files in chunk? And how to do that? Thanks!</p>
  
  <p>== EDIT ==</p>
  
  <p>After some more testing, it seems that memory limit only applies to <code>f.read()</code>, while writing to a large file is okay. Reading files in chunks solved my issue, but I really like the <code>compose()</code> function as @Ian-Lewis pointed out. Thanks!</p>