title: Transfer large number of large files to s3
body: <p>I am transferring around 31 TB of data that consists of 4500 files, file sizes range from 69MB to 25GB, from a remote server to a s3 bucket. I am using <code>s4cmd put</code> to do this and put it in a bash script <code>upload.sh</code>:</p>
  
  <pre><code>#!/bin/bash
  
  FILES="/path/to/*.fastq.gz"
  for i in $FILES
  do
      echo "$i"
      s4cmd put --sync-check -c 10 $i s3://bucket-name/directory/
  done
  </code></pre>
  
  <p>Then I use <code>qsub</code> to submit the job:</p>
  
  <pre><code>qsub -cwd -e error.txt -o output.txt -l h_vmem=10G -l mem_free=8G -l m_mem_free=8G -pe smp 10 upload.sh
  </code></pre>
  
  <p>This is taking way too long - it took 10 hours to upload ~20 files. Can someone suggest alternatives or modifications to my command? </p>
  
  <p>Thanks!</p>