Processing objects of different classes in Python in parallel
Question: I have 6 different classes.

In main, I am creating one object of each class.

I want to call `eachObject.processFiles()` of each object parallely.

The processFiles method processes the object files by reading them,
manipulates the data, and save it in persistent object. Each processFiles call
takes around 5 minutes. If it is done sequentially then main is taking around
30 min.

I am right now doing it sequentially, but I want to speed up the 6 objects'
processing their files in minimum cpu time. All objects are independent of
each other, and I think multithreaded way would be efficient. But I haven't
done multi threading any before. So wanted to know if it would be safe to do
that, and how to do that. A code snippet would help.

How can I do it in parallel?

    
    
    class system(object):
      def __init__(self, leNameList):
        self.files = fileNameList
    
      def processFiles(self):
        self.feeds= self.readFiles()
        self.processFeeds()
    
    class A(system):
      def processFeeds(self):
        """ process the feed 
             in A way """
    
    class B(system):
      def processFeeds(self):
        """ process the feed 
             in B way """
    
    def main():
      aObj = A(fileList)
      bObj = B(fileList2)
      aObj.processFiles()
      bObj.processFiles() 
    


Answer: Use the
[`multiprocessing`](https://docs.python.org/2/library/multiprocessing.html)
module.

    
    
    import multiprocessing
    
    class system(object):
      def __init__(self, fileNameList):
        self.files = fileNameList
    
      def processFiles(self):
        # self.feeds= self.readFiles()
        return self.processFeeds()
    
    class A(system):
      def processFeeds(self):
        return ["A feed", "example data"] + self.files
    
    class B(system):
      def processFeeds(self):
        return ["B feed", "hello world"] + self.files
    
    def process_file_task(processor):
        return processor.processFiles()
    
    def main():
      aObj = A(["a"])
      bObj = B(["b"])
      data = multiprocessing.Pool().map(process_file_task, [aObj, bObj])
      print(data)
    

Here I've populated your code with example data, so that you can test drive
this solution out of the box, but the gist of it is:

  * Instead of altering global variables or object state, return the data you generate.
  * Use `multiprocessing.Pool().map` to execute the functions in different processes and return each result in order.

The drawback to this is that `multiprocessing` can be a bit finicky to work
with at times, and you have to ensure that everything passing process
boundaries can be pickled.

* * *

One more thing to check in your code is the file lists. If they have a lot of
files in common, you should separate the process of file reading out and
ensure that the same file doesn't have to be read twice.

