Compare multiple file in python
Question: i have a set of directories with `n` number of files, I need to compare each
of those files (within one directory) and find if there is any difference in
them. I tried `filecmp` and `difflib` but they only support two files.

Is there anything else I can do to compare/diff the files ?

This file contain hosts name

    
    
    --------------------------------
    Example :- Dir -> Server.1
                        |-> file1
                        |-> file2
                        |-> file3
    
    
    file1 <- host1 
             host2
             host3
    
    file2 <- host1 
             host2 
             host3 
             host4
    
    file3 <- host1 
             host2 
             host3
    


Answer: I thought I'd share how combining the md5 hash compare with os.path.walk() can
help you ferret out all the duplicates in a directory tree. The larger the
number of directories and files gets, the more helpful it might be to first
sort files by size to rule out any files that can't duplicates because they
are of different size. Hope this helps.

    
    
    import os, sys
    from hashlib import md5
    
    nonDirFiles = []
    
    def crawler(arg, dirname, fnames):
        '''Crawls directory 'dirname' and creates global 
        list of paths (nonDirFiles) that are files, not directories'''
        d = os.getcwd()
        os.chdir(dirname)
    
        global nonDirFiles
        for f in fnames:
            if not os.path.isfile(f):
    
                continue
            else:       
                nonDirFiles.append(os.path.join(dirname, f))
        os.chdir(d)
    
    def startCrawl():
        x = raw_input("Enter Dir: ")
        print 'Scanning directory "%s"....' %x
        os.path.walk(x, crawler, nonDirFiles)
    
    def findDupes():
        dupes = []
        outFiles = []
        hashes = {}
        for fileName in nonDirFiles:
            print 'Scanning file "%s"...' % fileName
            f = file(fileName, 'r')
            hasher = md5()
            data = f.read()
            hasher.update(data)
            hashValue = hasher.digest()
    
            if hashes.has_key(hashValue):
    
                dupes.append(fileName)
            else:
                hashes[hashValue] = fileName
    
        return dupes
    
    if __name__ == "__main__":
        startCrawl()
        dupes = findDupes()
        print "These files are duplicates:"
        for d in dupes:print d
    

