Trying to multiprocess a function requiring a list argument in python
Question: My problem is that I'm trying to pass a `list` as a variable to a function,
and I'd like to mutlti-thread the function processing. I can't seem to use
`pool.map` because it only accepts iterables. I can't seem to use `pool.apply`
because it seems to block the pool while it works, so I don't really
understand how it allow mutli-threading at all (admittedly, I don't seem to
understand anything about multi-threading). I tried `pool.apply_async`, but
the program finishes in seconds, and only appears to process about 20000 total
computations. Here's some psuedo-code for it.

    
    
    import MySQLdb
    from multiprocessing import Pool
    
    def some_math(x, y):
        f(x[1], x[2], y[1], y[2])
        return f
    
    def distance(x):
        x_distances = []
        for y in all_y:
            distance = some_math(x, y)
            if distance > 1000000:
                continue
            else:
                x_distances.append(x[0], y[0],distance)
            mysql.executemany(sql_update, x_distances)
            mydb.commit()
    
    all_x = []
    all_y = []
    sql_x = 'SELECT id, lat, lng FROM table'
    sql_y = 'SELECT id, lat, lng FROM table'
    sql_update = 'INSERT INTO distances (id_x, id_y, distance) VALUES (%s, %s, %S)'
    
    cursor.execute(sql_x)
    all_x = cursor.fetchall()
    
    cursor.execute(sql_y)
    all_y = cursor.fetchall()
    
    p = Pool(4)
    for x in all_x:
        p.apply_async(distance, x)
    

OR, if using map:

    
    
    p = Pool(4)
    for x in all_x:
        p.map(distance, x)
    

The error returns: Processing A for distances...

    
    
    Traceback (most recent call last):
      File "./distance-house.py", line 94, in <module>
        p.map(range, row)
      File "/usr/lib/python2.7/multiprocessing/pool.py", line 251, in map
        return self.map_async(func, iterable, chunksize).get()
      File "/usr/lib/python2.7/multiprocessing/pool.py", line 558, in get
        raise self._value
    TypeError: 'float' object has no attribute '__getitem__'
    

I am trying to multi-thread a long computation - calculating a the distance
between something like 10,000 points on a many-to-many basis. Currently, the
process is taking several days, and I figure that multiprocessing the results
could really improve the efficiency. I'm all ears for suggestions.


Answer: You can use `pool.map`:

    
    
    p = Pool(4)
    p.map(distance, all_x)
    

as per the first example in the
[doc](https://docs.python.org/2/library/multiprocessing.html). It will do the
iteration for you!

