More functional os.walk
Question: Since I need to do many traversals of directories, which some complex
filtering, I thought to create a wrapper around os.walk.

Which is something like this:

    
    
    def fwalk(root, pred_dir, pred_files, walk_function=walk):
        """Wrapper function around the standard os.walk, that filter out
        the directories visited using a filtering predicate
        """
    
        for base, dirs, files in walk_function(root):
            # ignore also the root directory when not needed, which is
            # actually more important than the subdirectories
            dirs = [d for d in dirs if pred_dir(path.join(base, d))]
            files = [f for f in files if pred_files(path.join(base, f))]
    
            if _ignore_dirs_predicate(base) and (dirs or files):
                yield base, dirs, files
    

Basically it behaves as os.walk, but takes two predicates to make it a bit
nicer to compose in higher-level functions. For example this will only go
through the python modules:

    
    
    ISA_PY = lambda f: f[-3:] == '.py'
    # I can make it a class or maybe even a module if it's better
    def walk_py(src):
        # should not be in the list
        return fwalk(src, _ignore_dirs_predicate, ISA_PY)
    

It also takes a walk function which for example can be just a dummy walk, used
for testing.

    
    
    def dummy_walk(_):
        test_dir = [
            ('/root/', ['d1, .git'], []),
            ('/root/d1', [], ['setup.py']),
            ('/root/test', [], ['test1.py']),
            ('/root/.git', [], [])
        ]
    
        # returns a function which  skips the parameter and return the iterator
        return iter(test_dir)
    

The problem now is that I find it very hard to trust this function, apart from
the some unit testing using the dummy walk is quite hard to make sure it's
correct.

Any suggestion about how I can improve this and make it nicer?


Answer: you need to modify dirs in place in order to avoid recursive traversal of the
removed directories. Use:

    
    
    dirs[:] = [d for d in dirs if pred_dir(path.join(base, d))]
    

this will remove the need to check `_ignore_dirs_predicate(base)` (and remove
the `NameError` caused by the use of `_ignore_dirs_predicate` instead of
`pred_dir`)

You should also rewrite `ISA_PY` to use `str.endswith()`

