Filtering a long list of files through a set of ignore patterns using iterators I have a backup job that walks across a huge directory with potentially millions of files. Python's os.walk() works just fine.

Now, I have implemented a feature to ignore files based in a black list of patterns. I just want it to behave like a .gitignore file.

Currently I do this:

import os
from fnmatch import fnmatch

some_dir = '/'
ignore_list = ['*.tmp', 'tmp/', '*.py']
for dirname, _, filenames in os.walk(some_dir):
    for filename in filenames:
        should_ignore = False
        for pattern in ignore_list:
            if fnmatch(filename, pattern):
                should_ignore = True
        if should_ignore:
            print 'Ignore', filename
            continue


It works, but I failed to write it in terms of iterators.

Can this routine be written in terms of python iterators? Can it be improved in terms of iterators or generators?
You can simplify the loop by using a generator expression like this:

for filename in filenames:
    if any(fnmatch(filename, pattern) for pattern in ignore_list):
        print 'Ignore', filename
        continue