This question resulted from a question of mine on stackexchange.I mount a windows share with mount -t cifs -o username=username,password=password,rw,nounix,iocharset=utf8,file_mode=0777,dir_mode=0777 //192.168.1.120/storage /mnt/storage, then have scripts operate on the mountpoint on a debian machine.The mountpoint /mnt/storage will contain a very rapidly growing number of files which are moved to a subdirectory in batches, where they are processed.My problem is that the moving is relatively slow and I fear that this is because I'm unable to just alter the filetable (simply change the information where the file is on the hdd).I'm currently using shutil.move(src,dst) in python, but also considered using os.rename(src,dst) or a subprocess using mv. Is my fear correct - if so is there a way to mount more efficiently?edit: I just went over the documentation of shutil.move() again and read this:If the destination is on the current filesystem, then os.rename() is used.Otherwise, src is copied (using shutil.copy2()) to dst and then removed.This sounds like it could be an issue, if it is not on the current filesystem, how do I know if I am on the current filesystemedit2: In case someone's interested in what I found out, I'll copy&paste me edit from stackoverflow here as well..I wrote a script to test the speed differences between the various methods of moving.First created 1x5GB (dd if=/dev/urandom of=/mnt/storage/source/test.file bs=100M count=50), then with 100x5MB (for i in {1..100}; do dd if=/dev/urandom of=/mnt/storage/source/file$i bs=1M count=5) and finally with 10000x5kB (for i in {1..100000}; do dd if=/dev/urandom of=/mnt/storage/source/file$i bs=1k count=5) from shutil import movefrom os import renamefrom datetime import datetimeimport subprocessimport osprint(Subprocess mv: for every file in directory..)s = datetime.now()for f in os.listdir(/mnt/storage/source/):    try:        subprocess.call([mv /mnt/storage/source/+str(f)+ /mnt/storage/mv],shell=True)    except Exception as e:        print(str(e))e = datetime.now()print(took {}.format(e-s)+\n)print(Subprocessmv : directory/*..)s = datetime.now()try:    subprocess.call([mv /mnt/storage/mv/* /mnt/storage/mvf],shell=True)except Exception as e:    print(str(e))e = datetime.now()print(took {}.format(e-s)+\n)print(shutil.move: for every file file in directory..)s = datetime.now()for f in os.listdir(/mnt/storage/mvf/):    try:            move(/mnt/storage/mvf/+str(f),/mnt/storage/move)    except Exception as e:        print(str(e))e = datetime.now()print(took {}.format(e-s)+\n)print(os.rename: for every file in directory..)s = datetime.now()for f in os.listdir(/mnt/storage/move/):    try:        rename(/mnt/storage/move/+str(f),/mnt/storage/rename/+str(f))    except Exception as e:        print(str(e))e = datetime.now()print(took {}.format(e-s)+\n)if os.path.isdir(/mnt/storage/rename_new):    rmtree('/mnt/storage/rename_new')print(os.rename & os.mkdir: rename source dir to destination & make new source dir..)s = datetime.now()rename(/mnt/storage/rename/,/mnt/storage/rename_new)os.mkdir(/mnt/storage/rename/)e = datetime.now()print(took {}.format(e-s)+\n)Which revealed that there's not that much of a difference.. The 5GB file was moved really fast, which tells me that the moving, by altering the file table works. Here are the results of the 10000*5kB files (It felt like the results depend on the current network workload. e.g. the first mv test took 2m 28s, than later with the same files 3m 22s, also was os.rename() the fastest method most of the times..):Subprocess mv: for every file in directory..took 0:02:47.665174Subprocessmv : directory/*..took 0:01:40.087872shutil.move: for every file file in directory..took 0:01:48.454184os.rename: for every file in directory..rename took 0:02:05.597933os.rename & os.mkdir: rename source dir to destination & make new source dir..took 0:00:00.005704
Properly mount a windows share to move files on the mountpoint
debian;mount;performance;cifs;move