title: Weird multiprocessing block importing Numba function
body: <h1>Environment</h1>
  
  <ul>
  <li>GNU/Linux (Fedora 25).</li>
  <li>Conda environment.</li>
  <li>Python 3.6.1.</li>
  <li>Numba 0.33.0 (np112py36_0).</li>
  </ul>
  
  <h1>Initial setup (works fine)</h1>
  
  <p>Two files <code>main.py</code> and <code>numbamodule.py</code>:</p>
  
  <h2><code>main.py</code></h2>
  
  <p>Which spawns 2 processes to run the <code>execute_numba</code> function.</p>
  
  <pre><code>import time
  from importlib import import_module
  from multiprocessing import Process
  
  
  def execute_numba(name):
      # Import the function
      importfunction = 'numbamodule.numba_function'
      module = import_module(importfunction.split('.')[0])
      function = getattr(module, importfunction.split('.')[-1])
      while True:
          print(str(name) + ' - executing Numba function...')
          # Execute the function
          function(10)
          time.sleep(0.1)
  
  
  if __name__ == '__main__':
      processes = [Process(target=execute_numba, args=(i,)) for i in range(2)]
      [p.start() for p in processes]
      time.sleep(1)
      [p.terminate() for p in processes]
  </code></pre>
  
  <h2><code>numbamodule.py</code></h2>
  
  <p>Which defines a simple function <code>numba_function</code>:</p>
  
  <pre><code>import numba
  
  
  @numba.jit()
  def numba_function(x):
      total = 0
      for i in range(x):
          total += i
      return total
  </code></pre>
  
  <p>I can run the <code>main.py</code> script and see both processes printing:</p>
  
  <pre><code>$ python main.py
  0 - executing Numba function...
  1 - executing Numba function...
  0 - executing Numba function...
  1 - executing Numba function...
  0 - executing Numba function...
  1 - executing Numba function...
  [...]
  </code></pre>
  
  <h1>Breaking it</h1>
  
  <p>The way I break it is a bit weird, but this is what I stumbled upon when trying to minimize a reproducible test case. Please, tell me if you can reproduce the same behavior too.</p>
  
  <p>In <code>main.py</code> I just add one of the proposed (bellow) imports after the last <code>Process</code> import (i.e.: uncomment one line and try):</p>
  
  <pre><code>import time
  from importlib import import_module
  from multiprocessing import Process
  
  #
  # Adding one of the import lines bellow results in a block...
  # (you may need to install the packages first in the virtual environment)
  #
  #import matplotlib
  #import Pyro4
  #import scipy
  #import dill
  
  
  def execute_numba(name):
  # [...]
  </code></pre>
  
  <p>Then one process may block at <code>execute_numba</code> function (in particular at the <code>import_module()</code> call):</p>
  
  <pre><code>$ python main.py 
  1 - executing Numba function...
  1 - executing Numba function...
  1 - executing Numba function...
  1 - executing Numba function...
  1 - executing Numba function...
  1 - executing Numba function...
  [...]
  </code></pre>
  
  <p>For me, <code>matplotlib</code> and <code>Pyro4</code> imports "work" the best. I cannot even get the block a 100% of the runs... :-/</p>
  
  <p>Note that I am simply adding a single import line, not actually using the package. Some other external imports result in a block as well, but I have found that the ones proposed above "work" best (block the most).</p>
  
  <h1>What is happening?</h1>
  
  <p>First of all, can you reproduce the same behavior? (specially interested in non-virtualized GNU/Linux machines)</p>
  
  <p>I don't know how to debug this or why could this be happening. Any ideas?</p>
  
  <p>The fact that adding one random <code>import xxx</code> triggers the block scares me and makes little sense to me. Could this be dependent on timing/delays and that is why some imports break it and some others do not?</p>
  
  <h1>Notes</h1>
  
  <ul>
  <li>As you can see there is no traceback, the process just blocks.</li>
  <li>If I remove the <code>import numba</code> and <code>@numba.jit</code> from <code>numbamodule.py</code>, then it will always work, so maybe it has something to do with Numba?</li>
  <li>I can reproduce the same behavior also with older Numba/Python versions. Tried with Numba 0.25.0 and 0.22.1 (both with Python 3.5.3).</li>
  </ul>
  
  <h1>Updates</h1>
  
  <ul>
  <li>2017-07-03: Just to make it clear, I am not looking for a workaround (I already have one in the real code). I am genuinely interested in knowing how to proceed in a case like this. Understand what is going on and learn how to debug and find the problem in order to report it if it is a broken package/build/environment. How would you proceed?</li>
  <li>2017-07-10: The block occurs in particular at the <code>import_module()</code> call.</li>
  <li>2017-07-11: <a href="https://github.com/numba/numba/issues/2431" rel="nofollow noreferrer">Numba issue acknowledged</a>.</li>
  </ul>