Python copy larger file too slow
Question: I am trying to copy a large file (> 1 GB) from hard disk to usb drive using
`shutil.copy`. A simple script depicting what I am trying to do is:-

    
    
    import shutil
    src_file = "source\to\large\file"
    dest = "destination\directory"
    shutil.copy(src_file, dest)
    

It takes only 2-3 min on linux. But the same file copy on same file takes more
that 10-15 min under Windows. Can somebody explain why and give some solution,
preferably using python code?

**Update 1**

Saved the file as test.pySource file size is 1 GB. Destinantion directory is
in USB drive. Calculated file copy time with ptime. Result is here:-

    
    
    ptime.exe test.py
    
    ptime 1.0 for Win32, Freeware - http://www.
    Copyright(C) 2002, Jem Berkes <jberkes@pc-t
    
    ===  test.py ===
    
    Execution time: 542.479 s
    

542.479 s == 9 min. I don't think `shutil.copy` should take 9 min for copying
1 GB file.

**Update 2**

Health of the USB is good as same script works well under Linux. Calculated
time with same file under windows native xcopy.Here is the result.

    
    
    ptime 1.0 for Win32, Freeware - http://www.pc-tools.net/
    Copyright(C) 2002, Jem Berkes <jberkes@pc-tools.net>
    
    ===  xcopy F:\test.iso L:\usb\test.iso
    1 File(s) copied
    
    Execution time: 128.144 s
    

128.144 s == 2.13 min. I have 1.7 GB free space even after copying test file.


Answer: Just to add some interesting information: WIndows does not like the tiny
buffer used on the internals of the shutil implementation.

I've quick tried the following:

  * Copied the original **shutil.py** file to the example script folder and renamed it to **myshutil.py**
  * Changed the 1st line to `import myshutil`
  * Edited the **myshutil.py** file and changed the **copyfileobj** from

> def copyfileobj(fsrc, fdst, length=16*1024):

to

> def copyfileobj(fsrc, fdst, length=16*1024*1024):

Using a 16 MB buffer instead of 16 KB caused a huge performance improvement.

Maybe Python needs some _tuning_ targeting Windows internal filesystem
characteristics?

