title: Improve performance of moving a growing, large number of files in a mounted folder
body: <p>This is my situation:</p>
  
  <p>A have a Windows network share, i mount with <code>mount -t cifs -o username=username,password=password,rw,nounix,iocharset=utf8,file_mode=0777,dir_mode=0777 //192.168.1.120/storage /mnt/storage</code></p>
  
  <p>This folder contains a very rapidly growing number of files of various size (few byte up to ~20MB). If not moved/deleted the number of files in this directory may exceed 10 million.</p>
  
  <p>I am required to move batches (in <code>move_script</code>) of files with a specific name (<code>*.fext</code>) from this directory to another directory (currently a subfolder in the directory <code>/mnt/storage/in_progress</code>).</p>
  
  <p>Then the script starts another script (<code>process_script</code>) which will process the files in <code>/mnt/storage/in_progress</code>. After <code>process_script</code> finished, the files are moved again by <code>move_script</code> to another subdirectory (<code>/mnt/storage/done</code>). The <em>move-process-move</em> continues until the source folder (<code>/mnt/storage</code>) contains no more files.</p>
  
  <p>Additional information of the process:</p>
  
  <ul>
  <li><p>the current bottleneck is the moving of the files (the files are moved a little faster than the files are created in the directory)</p>
  
  <pre><code>if len(os.listdir("/mnt/storage") &gt;= batch_size:
      i = 0
      for f in os.listdir("/mnt/storage"):
          if f.endswith(".fext"):
              move("/mnt/storage/+"f","/mnt/storage/in_progress"
              i+=1
          if i==batch_size:
              break
  </code></pre></li>
  <li><p>the script moving/starting the processing of the files, waits for the processing to finish</p></li>
  <li><p>processing of the files in <code>/mnt/storage/in_progress</code> is fastest with batches of 1k-2k files.</p></li>
  <li><p>I have tried to have the number of files that are moved growing. First move 1k, then if the number of files in the source directory is growing, double number of files that are moved.. This slows down the processing of the files in <code>process_script</code>, but helps to keep up with the "file-generator".. </p></li>
  <li><p>I considered to simply rename the subdirectory <code>/mnt/storage/in_progress</code>, after <code>process_script</code> finished to <code>"/mnt/storage/done"+i_counter</code> and create a new <code>/mnt/storage/in_progress</code>. I assume this would half the move-time in the script.</p></li>
  </ul>
  
  <p>I'll need to speed up the process, in order to keep up with the file-generator. How could I increase the performance of this move operation? </p>
  
  <p>I'm open for any suggestion and willing to completely change my current approach.</p>
  
  <p>edit: The scripts run on debian wheezy, so I theoretically could use a subprocess issuing <code>mv</code>, but I have no clue how reasonable that would be.</p>
  
  <p>==========================================</p>
  
  <p>edit2:
  I wrote a script to test the speed differences between the various methods of moving.
  First created 1x5GB (<code>dd if=/dev/urandom of=/mnt/storage/source/test.file bs=100M count=50</code>), then with 100x5MB (<code>for i in {1..100}; do dd if=/dev/urandom of=/mnt/storage/source/file$i bs=1M count=5</code>) and finally with 10000x5kB (<code>for i in {1..100000}; do dd if=/dev/urandom of=/mnt/storage/source/file$i bs=1k count=5</code>) </p>
  
  <pre><code>from shutil import move
  from os import rename
  from datetime import datetime
  import subprocess
  import os
  
  print("Subprocess mv: for every file in directory..")
  s = datetime.now()
  for f in os.listdir("/mnt/storage/source/"):
      try:
          subprocess.call(["mv /mnt/storage/source/"+str(f)+" /mnt/storage/mv"],shell=True)
      except Exception as e:
          print(str(e))
  e = datetime.now()
  print("took {}".format(e-s)+"\n")
  
  print("Subprocessmv : directory/*..")
  s = datetime.now()
  try:
      subprocess.call(["mv /mnt/storage/mv/* /mnt/storage/mvf"],shell=True)
  except Exception as e:
      print(str(e))
  e = datetime.now()
  print("took {}".format(e-s)+"\n")
  
  
  print("shutil.move: for every file file in directory..")
  s = datetime.now()
  for f in os.listdir("/mnt/storage/mvf/"):
      try:    
          move("/mnt/storage/mvf/"+str(f),"/mnt/storage/move")
      except Exception as e:
          print(str(e))
  e = datetime.now()
  print("took {}".format(e-s)+"\n")
  
  print("os.rename: for every file in directory..")
  s = datetime.now()
  for f in os.listdir("/mnt/storage/move/"):
      try:
          rename("/mnt/storage/move/"+str(f),"/mnt/storage/rename/"+str(f))
      except Exception as e:
          print(str(e))
  e = datetime.now()
  print("took {}".format(e-s)+"\n")
  
  
  if os.path.isdir("/mnt/storage/rename_new"):
      rmtree('/mnt/storage/rename_new')
  print("os.rename &amp; os.mkdir: rename source dir to destination &amp; make new source dir..")
  s = datetime.now()
  rename("/mnt/storage/rename/","/mnt/storage/rename_new")
  os.mkdir("/mnt/storage/rename/")
  e = datetime.now()
  print("took {}".format(e-s)+"\n")
  </code></pre>
  
  <p>Which revealed that there's not that much of a difference.. The 5GB file was moved really fast, which tells me that the moving, by altering the file table works. Here are the results of the 10000*5kB files (It felt like the results depend on the current network workload. e.g. the first <code>mv</code> test took 2m 28s, than later with the same files 3m 22s, also was <code>os.rename()</code> the fastest method most of the times..):</p>
  
  <pre><code>Subprocess mv: for every file in directory..
  took 0:02:47.665174
  
  Subprocessmv : directory/*..
  took 0:01:40.087872
  
  shutil.move: for every file file in directory..
  took 0:01:48.454184
  
  os.rename: for every file in directory..
  rename took 0:02:05.597933
  
  os.rename &amp; os.mkdir: rename source dir to destination &amp; make new source dir..
  took 0:00:00.005704
  </code></pre>