benchmarks: does python have a faster way of walking a network folder?
Question: I need to walk through a folder with approximately ten thousand files. My old
vbscript is very slow in handling this. Since I've started using Ruby and
Python since then, I made a benchmark between the three scripting languages to
see which would be the best fit for this job.

The results of the tests below on a subset of 4500 files on a shared network
are

    
    
    Python: 106 seconds
    Ruby: 5 seconds
    Vbscript: 124 seconds
    

That Vbscript would be slowest was no surprise but I can't explain the
difference between Ruby and Python. Is my test for Python not optimal? Is
there a faster way to do this in Python?

The test for thumbs.db is just for the test, in reality there are more tests
to do.

I needed something that checks every file on the path and doesn't produce too
much output to not disturb the timing. The results are a bit different each
run but not by much.

    
    
    #python2.7.0
    import os
    
    def recurse(path):
      for (path, dirs, files) in os.walk(path):
        for file in files:
          if file.lower() == "thumbs.db":
            print (path+'/'+file)
    
    if __name__ == '__main__':
      import timeit
      path = '//server/share/folder/'
      print(timeit.timeit('recurse("'+path+'")', setup="from __main__ import recurse", number=1))
    
    
    
    'vbscript5.7
    set oFso = CreateObject("Scripting.FileSystemObject")
    const path = "\\server\share\folder"
    start = Timer
    myLCfilename="thumbs.db"
    
    sub recurse(folder)
      for each file in folder.Files
        if lCase(file.name) = myLCfilename then
          wscript.echo file
        end if
      next
      for each subfolder in folder.SubFolders
        call Recurse(subfolder)
      next
    end Sub
    
    set folder = oFso.getFolder(path)
    recurse(folder)
    wscript.echo Timer-start
    
    
    
    #ruby1.9.3
    require 'benchmark'
    
    def recursive(path, bench)
      bench.report(path) do
        Dir["#{path}/**/**"].each{|file| puts file if File.basename(file).downcase == "thumbs.db"}
      end
    end
    
    path = '//server/share/folder/'
    Benchmark.bm {|bench| recursive(path, bench)}
    

EDIT: since i suspected the print caused a delay i tested the scripts with
printing all 4500 files and also printing none, the difference remains, R:5
P:107 in the first case and R:4.5 P:107 in the latter

EDIT2: based on the answers and comments here a Python version that in some
cases could run faster by skipping folders

    
    
    import os
    
    def recurse(path):
      for (path, dirs, files) in os.walk(path):
        for file in files:
          if file.lower() == "thumbs.db":
            print (path+'/'+file)
    
    def recurse2(path):
        for (path, dirs, files) in os.walk(path):
            for dir in dirs:
                if dir in ('comics'):
                    dirs.remove(dir)
            for file in files:
                if file.lower() == "thumbs.db":
                    print (path+'/'+file)
    
    
    if __name__ == '__main__':
      import timeit
      path = 'f:/'
      print(timeit.timeit('recurse("'+path+'")', setup="from __main__ import recurse", number=1)) 
    #6.20102692
      print(timeit.timeit('recurse2("'+path+'")', setup="from __main__ import recurse2", number=1)) 
    #2.73848228
    #ruby 5.7
    


Answer: The Ruby implementation for `Dir` is in C (the file `dir.c`, according to
[this documentation](http://www.ruby-doc.org/core-1.9.3/Dir.html)). However,
the Python equivalent is implemented [in
Python](http://hg.python.org/cpython/file/abe8a2908f08/Lib/os.py#l209).

It's not surprising that Python is less performant than C, but the approach
used in Python gives a little more flexibility - for example, you could skip
entire subtrees named e.g. `'.svn'`, `'.git'`, `'.hg'` while traversing a
directory hierarchy.

Most of the time, the Python implementation is fast enough.

**Update:** The skipping of files/subdirs doesn't affect the traversal _rate_
at all, but the overall time taken to process a directory tree could certainly
be reduced because you avoid having to traverse potentially large subtrees of
the main tree. The time saved is of course proportional to how much you skip.
In your case, which looks like folders of images, it's unlikely you would save
much time (unless the images were under revision control, when skipping
subtrees owned by the revision control system might have some impact).

**Additional update:** Skipping folders is done by changing the `dirs` value
in place:

    
    
    for root, dirs, files in os.walk(path):
        for skip in ('.hg', '.git', '.svn', '.bzr'):
            if skip in dirs:
                dirs.remove(skip)
            # Now process other stuff at this level, i.e.
            # in directory "root". The skipped folders
            # won't be recursed into.
    

