title: How does Parallel Processing vs. running several python-kernels work in Python?
body: <p>I have a user-defined function (UDF) that takes a while to run - like a day or so. Additionally, that UDF needs to be applied repeatedly to different data sets. To be specific, my function opens a data set, edits it, then saves the data set. It doesn't return anything.</p>
  
  <p>What are the best ways to speed it up? Would it be better if I open up like 3 Jupyter Notebooks and run 3 at a time - each on a different data set? Or would it be better to do parallel processing using the <strong><code>multiprocessing</code></strong> package?</p>
  
  <p>I tried following the guide here: <a href="https://www.machinelearningplus.com/python/parallel-processing-python/" rel="nofollow noreferrer">https://www.machinelearningplus.com/python/parallel-processing-python/</a> for parallel processing but it is not working. Not wanting to edit my original UDF, I basically just created another user-defined function called <code>loop()</code> like so:</p>
  
  <pre><code>def loop( i ):
       UDF( i, &lt;...some other args that are not dynamic...&gt; )
  </code></pre>
  
  <p>Then I tried to do an asynchronous parallel processing using a <code>.map_async()</code>-method like so. Let's say I have 10 sets of data that I need to go over and I stored the names of my dataset in a list called <code>datalist</code>:</p>
  
  <pre><code>import multiprocessing as mp
  
  pool = mp.Pool( mp.cpu_count() )
  
  pool.map_async( loop, [i for i in datalist] )
  </code></pre>
  
  <p>All I got in return after executing my code is:</p>
  
  <pre><code>&lt;multiprocessing.pool.MapResult at 0x18cb2975160&gt;
  </code></pre>
  
  <p>What am I doing wrong? And which way is better? I am really new at all this programming stuff. Sorry if my terminology is off as I am self teaching.</p>