Fast Upload, Slow Download on Wireless-AC WAN over SFTP Title says most of it.  I use ES File Explorer, but the issue seems to exist in any file manager.  Copying files to the SFTP server nets 20-30MB/s, whereas copying files from is ~400-500kB/s.

I gather this may have something to do with Android treating these copy operations as a background download, but that's mostly supposition.
Maybe the site/computer you are downloading from have a upload limit. but when you push files (you upload) the other side is downloading, so it less of a problem for them.