Performance problems in Python: os.walk() + filecmp.dircmp().report_full_closure()
<p>I currently have a need to compare directories after incremental data migrations occur.  I wrote a python script to iterate through a list of source/destinations, perform the incremental copy from source to destination, then immediately compare the number of files and folders in each directory.  To do this comparison, we very simply use:</p>

<blockquote>
  <p>for (path, dirs, files) in os.walk(destd):</p>
  
  <p>destFileCount += len(files)</p>
  
  <p>destDirCount += len(dirs)</p>
</blockquote>

<p>If the number of files/dirs returned are different, we call another section of code to see what exactly is different.  To do that, we run the following and send the output to a file:</p>

<blockquote>
  <p>filecmp.dircmp(sourced, destd).report_full_closure()</p>
</blockquote>

<p>We use the report_full_closure piece as I'm not aware of another way to do a recursive comparison.  The script then searches the resulting file for lines starting with "only in" and prints them to the screen, effectively showing us the differences.</p>

<p>However inefficient, this works like a charm on directories with under 90,000 files or so but once we hit that upper limit the script becomes sluggish to the extent that it isn't feasible to use it for this purpose.  I suppose my questions can be separated into the following:</p>

<ol>
<li><p>Am I making a logical error in using both of these modules [os.walk + filecmp.dircmp().report_full_closure()]? i.e., am I really saving time being able to skip the filecmp, or should I just only do the filecmp and skip the file/dir count altogether?</p></li>
<li><p>Is there any way to combine these two functions by sort of 'caching' the files from one for use in the other?</p></li>
<li><p>Is there a quicker way to perform either of these functions?  I've searched high and low, so I'm guessing there is not.</p></li>
</ol>

<p>I really appreciate your thoughts on this matter.  This script has morphed and grown considerably so please forgive me if the answer is extremely obvious...
Thank you,
M</p>
