python write files and maintain folder structure
Question: I'm working on a script that will read files from one location, manipulate the
data, and then write the output to a different location. At the command line
the user will use a -p to specify a top-level folder and then the script will
recurse through there and find all the files. I'm doing this using glob right
now and the reading of the file is fine.

But I also want the user to specify an output folder to write the files into
and I want to maintain the folder structure of the input path.

    
    
    for eachFile in glob(args.path + "/*/*.json"): <- this seems dangerous. Better way?
      # do something to the json file
    
      # output the modified data to its new home
      #outfile = os.path.join(args.output, os.path.dirname(eachFile), eachFile) <- doesn't work
      outfile = os.path.join(args.putout, os.path.dirname(eachFile)[1:], eachFile)
    

That last line is the best I've done but it has the problem of assuming this
is being run on a posix machine when it strips off the "/" at the front of the
directory. Also, let's say I pass in an input path of ~/Documents/2014 and an
output of /tmp. The files will be written to
/tmp/Users/myusername/Documents/2014/blah/whatever.json.

This seems like a fairly common use case so I'm surprised that I haven't found
someone else that needed to do this or a simple module that does it easily.
Any suggestions?


Answer: Here is a script that does close to what you need. The key thing here, is
instead of glob, you needed os.walk because you want to drill down through the
directory structure. You'll need to add sanity checks, but it's a good start.

    
    
    # Recurse and process files.
    import os
    import sys
    from fnmatch import fnmatch
    import shutil
    
    
    def process(src_dir, dst_dir, pattern='*'):
        """Iterate through src_dir, processing all files that match pattern and
        store them, including their parent directories in dst_dir.
        """
        assert src_dir != dst_dir, 'Source and destination dir must differ.'
        for dirpath, dirnames, filenames in os.walk(src_dir):
            # Filter out files that match pattern only.
            filenames = filter(lambda fname: fnmatch(fname, pattern), filenames)
    
            if filenames:
                dir_ = os.path.join(dst_dir, dirpath)
                os.makedirs(dir_)
                for fname in filenames:
                    in_fname = os.path.join(dirpath, fname)
                    out_fname = os.path.join(dir_, fname)
    
                    # At this point, the destination directory is created and you
                    # have a valid input / output filename, so you'd call your
                    # function to process these files.  I just copy them :D
                    shutil.copyfile(in_fname, out_fname)
    
    if __name__ == '__main__':
        process(sys.argv[1], sys.argv[2], '*.txt')
    

