Make a Progress Counter in Concurrent Python
Question: 
    from multiprocessing import Process, Pool
    from bs4 import BeautifulSoup
    
    alreadyProcessedFile = 0.0
    processLength = 0.0
    
    def convertedFile(filePath):
    
        fileName = os.path.basename(filePath).split('.')[0]
    
        print("Total: "+str(processLength)+", current: "+str(alreadyProcessedFile)
              + ", percentage: "+ str((alreadyProcessedFile+1.0)/(processLength+1)))
    
        #business logic
    
        print("One file is saved at the location: "+r'E:\xxx\SECProject\ProcessedSEC10KFiles\2012QTR1/'+fileName+'-finalDoc.txt')
    
    def getFilePath(path):
        return glob.glob(os.path.join(path, '*.txt'))
    
    if __name__ == '__main__':
        s = r"E:\xxx\SECProject\SEC10KFiles\2012QTR1"
        fileList = getFilePath(s)
        processLength = len(fileList)
        p = Pool(40)
        p.map(convertedFile, fileList)
    

This is basically a shortened version of the python program. I don't have a
wrapper class outside and I want to have a counter inside to keep track of the
progress (so I know when the program will stop). However the program (`def
covertedFile()` have a problem accessing the changed variables:
`alreadyProcessedFile` and `processLength`)

I know concurrency might have problems with shared state, but how can you
accomplish a counter without some degree of shared state?? And why can't the
concurrent function access the variable outside of it?

* * *

Also as weird as it may sound, this concurrent version program can't process
all files (about 5000 of them) under one directory. I wonder if the program is
shut down while some processes are still running. Is this a possibility?


Answer: `multiprocessing` creates a _copy_ of the memory for each process at the time
you start the new processes. Sharing state via module-level globals doesn't
work because every process gets it's own "snapshot" of the globals at the time
you fork the process. Changing one processes' globals doesn't magically
propagate to other processes.

If you need to share state between processes, you need to use one of the tools
that `multiprocessing` gives to you. You can share c types via `Value` or
`Array`, or use `Manager` to share arbitrary data types. See [sharing state
between
processes](https://docs.python.org/2/library/multiprocessing.html#sharing-
state-between-processes) in the docs for details and examples.

