title: Fastest way to sync two Amazon S3 buckets
body: <p>I have a S3 bucket with around 4 million files taking some 500GB in total. I need to sync the files to a new bucket (actually changing the name of the bucket would suffice, but as that is not possible I need to create a new bucket, move the files there, and remove the old one).</p>
  
  <p>I'm using AWS CLI's <code>s3 sync</code> command and it does the job, but takes a lot of time. I would like to reduce the time so that <strong>the dependent system downtime is minimal</strong>.</p>
  
  <p>I was trying to run the sync both from my local machine and from <code>EC2 c4.xlarge</code> instance and there isn't much difference in time taken. </p>
  
  <p>I have noticed that the time taken can be somewhat reduced when I split the job in multiple batches using <code>--exclude</code> and <code>--include</code> options and run them in parallel from separate terminal windows, i.e.</p>
  
  <pre><code>aws s3 sync s3://source-bucket s3://destination-bucket --exclude "*" --include "1?/*" 
  aws s3 sync s3://source-bucket s3://destination-bucket --exclude "*" --include "2?/*" 
  aws s3 sync s3://source-bucket s3://destination-bucket --exclude "*" --include "3?/*" 
  aws s3 sync s3://source-bucket s3://destination-bucket --exclude "*" --include "4?/*" 
  aws s3 sync s3://source-bucket s3://destination-bucket --exclude "1?/*" --exclude "2?/*" --exclude "3?/*" --exclude "4?/*"
  </code></pre>
  
  <p>Is there anything else I can do speed up the sync even more? Is another type of <code>EC2</code> instance more suitable for the job? Is splitting the job into multiple batches a good idea and is there something like 'optimal' number of <code>sync</code> processes that can run in parallel on the same bucket?</p>
  
  <p><strong>Update</strong></p>
  
  <p>I'm leaning towards the strategy of syncing the buckets before taking the system down, do the migration, and then sync the buckets again to copy only the small number of files that changed in the meantime. However running the same <code>sync</code> command even on buckets with no differences takes a lot of time.</p>