title: Concatenate s3 files when using AWS Firehose
body: <p>I have an <a href="http://aws.amazon.com/documentation/kinesis/" rel="noreferrer">AWS Kinesis</a> Firehose stream putting data in s3 with the following config:</p>
  
  <pre><code>S3 buffer size (MB)*       2
  S3 buffer interval (sec)*  60
  </code></pre>
  
  <p>Everything works fine. The only problem is that Firehose creates one s3 file for every chunk of data. (In my case, one file every minute, as in the screenshot). Over time, this is a lot of files: 1440 files per day, 525k files per year.</p>
  
  <p><a href="https://i.stack.imgur.com/dVOdx.png" rel="noreferrer"><img src="https://i.stack.imgur.com/dVOdx.png" alt="enter image description here"></a></p>
  
  <p>This is hard to manage (for example if I want to copy the bucket to another one I would need to copy every single file one by one and this would take time).</p>
  
  <p>Two questions:</p>
  
  <ul>
  <li>Is there a way to tell Kinesis to group/concatenate old files together. (Eg, files older than 24 hours are grouped into chunks one one day). </li>
  <li>How is COPY redshift performance affected when <code>COPY</code>ing from a plethora of s3 files versus just a few ? I haven't measured this precisely, but in my experience performance with a lot of small files is quite worse. From what I can recall, when using big files, a COPY of about 2M rows is about ~1minute. 2M rows with lots of small files (~11k files), it takes up to 30minutes.</li>
  </ul>
  
  <p>My two main concerns are:</p>
  
  <ul>
  <li>Better redshift COPY performance (from s3)</li>
  <li>Easier overall s3 file management (backup, manipulation of any kind)</li>
  </ul>