What's the best way to download data from the SRA? Is it really this slow? I'm trying to download three WGS datasets from the SRA that are each between 60 and 100GB in size.  So far I've tried:


Fetching the .sra files directly from NCBI's ftp site
Fetching the .sra files directly using the aspera command line (ascp)
Using the SRA toolkit's fastqdump and samdump tools


It's excruciatingly slow.  I've had three fastqdump processes running in parallel now for approximately 18 hours.  They're running on a large AWS instance in the US east (Virginia) region, which I figure is about as close to NCBI as I can get.  In 18 hours they've downloaded a total of 33GB of data.  By my calculation that's ~500kb/s.  They do appear to still be running - the fastq files continue to grow and their timestamps continue to update.

At this rate it's going to take me days or weeks just to download the datasets.  Surely the SRA must be capable of moving data at higher rates that this?  I've also looked, and unfortunately the datasets I'm interested have not been mirrored out to ENA or the Japanese archive, so it looks like I'm stuck working with the SRA.

Is there a better way to fetch this data that wouldn't take multiple days?
Proximity to NCBI may not necessarily give you the fastest transfer speed. AWS may be deliberately throttling the Internet connection to limit the likelihood that people will use it for undesirable things. There's a chance that a home network might be faster, but you're likely to get the fastest connection to NCBI by using an academic system that is linked to NCBI via a research network.

Another possibility is using Aspera for downloads. This is unlikely to help if bandwidth is being throttled, but it might help if there's a bit of congestion through the regular methods:

https://www.ncbi.nlm.nih.gov/public/

NCBI also has an online book about best practises for downloading data from their servers.

On a related note, just in case someone sees this and EBI/ENA is an option, there's a great guide for how to do file transfer using Aspera on the EBI web site:

https://www.ebi.ac.uk/ena/browse/read-download#downloading_files_aspera


  Your command should look similar to this on Unix:


ascp -QT -l 300m -i &lt;aspera connect installation directory&gt;/etc/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:&lt;file or files to download&gt; &lt;download location&gt;


In my case, I've just started downloading some files from a MinION sequencing run. The estimated completion time via standard FTP was 12 hours for about 32GB of data; ascp has reduced that estimated download time to about an hour. Here's the command I used for downloading:

ascp -QT -l 300m -i ~/.aspera/connect/etc/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:/vol1/ERA932/ERA932268/oxfordnanopore_native/20160804_Mock.tar.gz .