What is the simplest and fastest way to transfer large file through a Windows network?
I have trouble transfering that huge file through the network. I've tried plain network copy, apache download, and ftp. Any method I tried end up failing when the amount of data transfered reach 2GB. The last time that I successfully transfered the file, it was through a usb attached external harddrive. But I want to perform this task routinely and preferably automatically.
Wonder what is the most pragmatic approach for this situation ?
I have copied many files over 20Gb using robocopy (as others have mentioned) but I'd avoid using the /MIR switch until you are sure you have got the copy doing what you want - since it will delete files as well as copy them.
SMB suffers from a one packet at a time limit so is often the slower way to copy files - you have the option to copy using push or pull. Personally, I prefer the push method (copy is initiated by the source).
eseutil /y source_file /d dest_file.
I highly recommend using the free utility RichCopy. It is multithreaded and can pause and resume file copy operations. I have had very good luck using it to transfer files between severs.
If you are copying over a dodgy connection. You can re run the download and it will go and find the files it didn't manage to get the first time.
As far as file copy utilities go, TeraCopy is a nice GUI-based one (not command line) that can queue lots of files, supports pausing and resuming, can dynamically change its buffer size to optimize speed, and can optionally replace Windows Explorer's default copy/move with its own.
MIR will MIRror the contents of one directory to another server. It will only copy files that have changed.
The most pragmatic solution to repeated shuffles of large SQL Server backup files is to use a third party backup compression product or SQL Server 2008 Enterprise Edition's built-in backup compression.
Are you copying the file over a LAN or through some WAN connection like ADSL? I assume it's a WAN because 20GB is not a big file to copy over a LAN. I copy many such files every day.
If it's a WAN connection then the way I do it is to use the Cygwin version of rsync.
I had network transfers fail at around the 2GB mark - it turned out to be a faulty NIC.
I use syncback ( http://www.2brightsparks.com/syncback/sbse.html ) daily for transferring files several times larger than yours. Never had a problem with it.
Its a bit late but I would recommend a 3rd party backup and restore application option. We use Red Gate SQL Backup (www.red-gate.com), it has compression and alternate location options in the GUI. I get compressions saving of 80% average - so you only transfer 20% of the actual db size. It also supports encryption so it can be used over a WAN without interception worries.
Its fully schedulable so can run automatically at a cycle of your choosing.
The GUI also allows you to configure and administer log shipping.
Free trial version available at the above.
I don't have experience with such a large file, but could you use robocopy or even xcopy with the /Z option that claims to be restartable. It seems like this is intended for large file copies where the network is not reliable.
Joking aside, netcat is likely the quickest way to transfer large files on a LAN. As no checksum is done you may wish to do a MD5 sum of the file before sending it and compare it to the MD5 sum of the received file.
I have used netcat a lot in this way, never seen it fail, never seen it fail to max out the network either..
Most of the time I copy that much data I am migrating a server share that users are attached to so I often use Replistor, www.emc.com/products/detail/software/replistor.htm. It will keep the source and dest directories in sync during the copy and afterward.
Compress the .bak file prior to copying. SQL .bak files compress way down if you're not using the fully allocated space in the data and log files.
Might eliminate the need for an alternative method for copying large files.
I don't think finding something to transfer faster is your problem, double check to make sure your target file system is NOT FAT, as others have said. Also, make sure the NICs on both sides have updated drivers and aren't otherwise acting flaky.
With that said, are you moving many many small files or just a few large ones? I've seen RAID controller problems when trying to move millions of tiny files.
I don't think you'll have a problem automating this once you figure out what is causing the failure. It may help to list more details about your hardware and any relevant errors you may see in event viewer.
Have you tried to use an eSATA connection to an external hard drive? Connections are smokin' fast (3 gigabit!) and should be able to transfer that file in no time!
What speed is your network card on the server, 10/100 or 10/100/1000? What does the server network bandwidth and switch look like when you are coping the file? What does the destination location's (server's?) network bandwidth look like when copying? Have you tried to teamed 2 NIC's together? Are the network card drivers up to date? Is the BIOS up to date?
There are many things that could be the issue for file transfers. Making sure the hardware drivers and BIOS are up to date can really make a difference.
The simplest and fastest way: external USB disks and walking.
My way: use rsync. If the copy fails, just restart it, and it will pick up where it left.
The other thing to check would be to see if the quota service is set up on the target server; I think it uses 2 GB as the default quota per user.
Not the answer you're looking for? Browse other questions tagged sql-server backup file-transfer files or ask your own question.
What is the best way to transfer a single large file over a high-speed, high-latency WAN link?
What is the fastest way to transfer file between home and servers?
What is fastest way to copy a sparse file? What method results in the smallest file?