title: Compare and sync two (huge) directories - consider only filenames
body: <p>I want to do a one-way sync in Linux between two directories. One contains files and the other one contains <em>processed</em> files but has the same directory structure and the same filenames but some files might be missing.</p>
  <p>Right now I am doing:</p>
  <pre><code>cd $SOURCE
  find * -type f | while read fname; do
      if [ ! -e &quot;$TARGET$fname&quot; ]
      then
          # process the file and copy it to the target. Create directories if needed.
      fi
  done
  </code></pre>
  <p>which works, but is painfully slow.
  <strong>Is there a better way to do this?</strong></p>
  <p>There are rughly 50.000.000 files, spread among directories and sub-directories. Each directory does not contain more than 255 files/subdirs.</p>
  <p>I looked at</p>
  <ul>
  <li><p><code>rsync</code>: seems like it always does size or timestamp comparison. This will result on every file being flagged as different since the processing takes some time and changes the file contents.</p>
  </li>
  <li><p><code>diff -qr</code>: could not figure out how to make it ignore file-sizes and content</p>
  </li>
  </ul>
  <h2>Edit</h2>
  <p>Valid assumptions:</p>
  <ul>
  <li>comparisons are being made solely on directory/file names</li>
  <li>we don't care about file metadata and/or attributes (eg, size, owner, permissions, date/time last modified, etc)</li>
  <li>we don't care about files that may reside in the target directory but without a matching file in the source directory. This is only partially true, but deletions from the source are rare and happen in bulk, so I will do a special case for that.</li>
  </ul>