Dynamically loading modules in Python (+ multi processing question)
<p>I am writing a Python package which reads the list of modules (along with ancillary data) from a configuration file.</p>

<p>I then want to iterate through each of the dynamically loaded modules and invoke a do_work() function in it which will spawn a new process, so that the code runs ASYNCHRONOUSLY in a separate process.</p>

<p>At the moment, I am importing the list of all known modules at the beginning of my main script - this is a nasty hack I feel, and is not very flexible, as well as being a maintenance pain.</p>

<p>This is the function that spawns the processes. I will like to modify it to dynamically load the module when it is encountered. The key in the dictionary is the name of the module containing the code:</p>

<pre><code>def do_work(work_info):
  for (worker, dataset) in work_info.items():
    #import the module defined by variable worker here...

    # [Edit] NOT using threads anymore, want to spawn processes asynchronously here...

    #t = threading.Thread(target=worker.do_work, args=[dataset])
    # I'll NOT dameonize since spawned children need to clean up on shutdown
    # Since the threads will be holding resources
    #t.daemon = True
    #t.start()
</code></pre>

<p><strong>Question 1</strong></p>

<p>When I call the function in my script (as written above), I get the following error:</p>

<blockquote>
  <p>AttributeError: 'str' object has no
  attribute 'do_work'</p>
</blockquote>

<p>Which makes sense, since the dictionary key is a string (name of the module to be imported).</p>

<p>When I add the statement:</p>

<p><strong>import worker</strong></p>

<p>before spawning the thread, I get the error:</p>

<blockquote>
  <p>ImportError: No module named worker</p>
</blockquote>

<p>This is strange, since the variable name rather than the value it holds are being used - when I print the variable, I get the value (as I expect) whats going on?</p>

<p><strong>Question 2</strong></p>

<p>As I mentioned in the comments section, I realize that the do_work() function written in the spawned children needs to cleanup after itself. My understanding is to write a clean_up function that is called when do_work() has completed successfully, or an unhandled exception is caught - is there anything more I need to do to ensure resources don't leak or leave the OS in an unstable state?</p>

<p><strong>Question 3</strong></p>

<p>If I comment out the t.daemon flag statement, will the code stil run ASYNCHRONOUSLY?. The work carried out by the spawned children are pretty intensive, and I don't want to have to be waiting for one child to finish before spawning another child.  BTW, I am aware that threading in Python is in reality, a kind of time sharing/slicing  - thats ok</p>

<p>Lastly is there a better (more Pythonic) way of doing what I'm trying to do?</p>

<p><strong>[Edit]</strong></p>

<p>After reading a little more about Pythons GIL and the threading (ahem - hack) in Python, I think its best to use separate processes instead (at least IIUC, the script can take advantage of multiple processes if they are available), so I will be spawning new processes instead of threads.</p>

<p>I have some sample code for spawning processes, but it is a bit trivial (using lambad functions). I would like to know how to expand it, so that it can deal with running functions in a loaded module (like I am doing above).</p>

<p>This is a snippet of what I have:</p>

<pre><code>def do_mp_bench():
    q = mp.Queue() # Not only thread safe, but "process safe"
    p1 = mp.Process(target=lambda: q.put(sum(range(10000000))))
    p2 = mp.Process(target=lambda: q.put(sum(range(10000000)))) 
    p1.start()
    p2.start()
    r1 = q.get()
    r2 = q.get()
    return r1 + r2
</code></pre>

<p>How may I modify this to process a dictionary of modules and run a do_work() function in each loaded module in a new process?</p>
