title: Python multiprocessing issue with arguments set/list completion of processes
body: <p>This other post is exactly, I think, regarding what i want to do.
  <a href="https://stackoverflow.com/questions/5442910/python-multiprocessing-pool-map-for-multiple-arguments">Python multiprocessing pool.map for multiple arguments</a></p>
  <p>How I am trying to implement is in my pseudo code:</p>
  <h1>called by another function in my code</h1>
  <pre><code>def find_similar(db, num, listofsets):
  
  #db is the sqlite3 database
  #num is a variable I need for the sql query
  #listofsets - is a list of sets.  each set is a set of strings
  
      threshold = 0.49
  
  
      similar_db_rows=[]
   
      for row in db.execute(&quot;SELECT thing1, thing2, thing3 FROM table WHERE num !={n};&quot;.format(n=num)):
      #thing3 is a long string, each value separated by a comma
          items = set(row[3].strip().split(','))
          for set_item in listofsets:
              sim_score = sim_function(set_item, items) 
              if sim_score&lt;threshold:
                  similar_db_rows.append(row)
      return similar_db_rows
  
  def sim_function(x,y):
  #x is a set, and y is a second set.  The function does some calculation
  and comparing, then returns a float value
  
      return float_value
  </code></pre>
  <p>This works.  What I was trying to do was use multiprocessing on the 2nd for loop.  Instead of iterating each set (as my list of sets can have a lot, and this is a major bottle neck) and calling the function, I wanted to use multiprocessing so that it would call the function for these sets, passing a set along with the second constant argument from the sqlquery, many at a time, then return the resulting number from each set into a list. After all of the sets have been processed, then I can use the items in that list to check if any item meets a threshold.</p>
  <p>I tried to use the <code>func_star</code> and <code>pool.map(func_star, itertools.izip(a_args, itertools.repeat(second_arg)))</code> by Sebestian AND the `parmap' by zeehio.  But for me, for example if I had 30 sets in the list, it was returning a list of results more than 30 times, each return it would check for similarity threshold, and appending rows, but never breaking out of this and I end up control Z'ing the whole thing.</p>
  <p>Below is an example of what I attempted, first using parmap:</p>
  <pre><code>def find_similar(db, num, listofsets):
  
  #db is the sqlite3 database
  #num is a variable I need for the sql query
  #listofsets - is a list of sets.  each set is a set of strings
  
  threshold = 0.49
  
  list_process_results=[]
  similar_db_rows=[]
   
  for row in db.execute(&quot;SELECT thing1, thing2, thing3 FROM table WHERE num !={n};&quot;.format(n=num)):
          items = set(row[3].strip().split(','))
          list_process_results = parmap.starmap(sim_function, zip(listofsets), items)
          print list_process_results
  
          if any(t &lt; threshold for t in list_process_results):
              #print &quot;appending a row&quot;
              similar_db_rows.append(row)
  return similar_db_rows
  </code></pre>
  <p>and <code>func_star</code>:</p>
  <pre><code>def func_star(a_b):
  &quot;&quot;&quot;Convert `f([1,2])` to `f(1,2)` call.&quot;&quot;&quot;
  return sim_function(*a_b)
  
  def find_similar(db, num, listofsets):
  pool = Pool()
  
  #db is the sqlite3 database
  #num is a variable I need for the sql query
  #listofsets - is a list of sets.  each set is a set of strings
  
  threshold = 0.49
  
  list_process_results=[]
  similar_db_rows=[]
   
  for row in db.execute(&quot;SELECT thing1, thing2, thing3 FROM table WHERE num !={n};&quot;.format(n=num)):
          items = set(row[3].strip().split(','))
          list_process_results=pool.map(func_star, itertools.izip(listofsets, itertools.repeat(items ) ))
          print list_process_results
  
          if any(t &lt; threshold for t in list_process_results):
              #print &quot;appending a row&quot;
              similar_db_rows.append(row)
  return similar_db_rows
  </code></pre>
  <p>The same is happening for me with both, it goes on forever, returning a list of the # I am expecting (a different set of values each time), &quot;appending a row&quot;, and never breaking out.</p>
  <p>Thanks for the help!!!  extra  is if multiprocessing can also be used for the results of the row query (the outer loop) but i will first conquer the inner loop</p>
  <p>To answer Dano question about <code>find_similar</code>()---
  I have another function that has a for loop.  Each iteration of this for loop calls <code>find_similar</code>.  When the resulting list is returned from <code>find_similar</code>, it prints the length of the list return, it then proceeds to finish the remainder of the loop, and go to the next for element.  After this for loop is finished, the function is over, and <code>find_similiar</code> is not called again.</p>