threads and concurrency allfiles (os path join(path,namefor pathdirsfiles in os walk(topdirfor name in filesdigest_map dict(digest_pool imap_unordered(compute_digest,allfiles, )digest_pool close(return digest_map try it out change the directory name as desired if _name_ =' _main_ 'digest_map build_digest_map("/users/beazley/software/python- "print(len(digest_map)in the examplea sequence of pathnames for all files in directory tree is specified using generator expression this sequence is then chopped up and farmed out to process pool using the imap_unordered(function each pool worker computes sha digest value for its files using the compute_digest(function the results are sent back to the master and collected into python dictionary although it' by no means scientific resultthis example gives percent speedup over single-process solution when run on the author' dual-core macbook keep in mind that it only makes sense to use process pool if the pool workers perform enough work to justify the extra communication overhead as general ruleit would not make sense to use pool for simple calculations such as just adding two numbers together shared data and synchronization normallyprocesses are completed isolated from each other with the only means of communication being queues or pipes howevertwo objects can be used to represent shared data underneath the coversthese objects use shared memory (via mmapto make access possible in multiple processes value(typecodearg argnlockcreates ctypes object in shared memory typecode is either string containing type code as used by the array module ( ' '' 'etc or type object from the ctypes module ( ctypes c_intctypes c_doubleetc all extra positional arguments arg arg argn are passed to the constructor for the given type lock is keyword-only argument that if set to true (the default) new lock is created to protect access to the value if you pass in an existing lock such as lock or rlock instancethen that lock is used for synchronization if is an instance of shared value created by valuethen the underlying value is accessed used value for examplereading value will get the value and assigning value will change the value rawvalue(typecodearg argnthe same as value except that there is no locking array(typecodeinitializerlockcreates ctypes array in shared memory typecode describes the contents of the array and has the same meaning as described for value(initializer is either an integer that sets the initial size of the array or sequence of items whose values and size are used to initialize the array lock is keyword-only argument with the same meaning as described for value(if is an instance of shared array created by arraythen you lib fl ff