Python multiprocessing with MySQL
Question: I've Googled this topic a lot, but cannot find a solution which fits my needs
:(

I have a MySQL DB with a table containing e-mail adresses (10,000+).

I would like to run a batch job on them every 5 minute.

So I'll guess Python is a good choice for retrieving the resultset from MySQL
and then call a command-line with the e-mail address' as arguments.

How do I do this the best way? I'm think of getting the entire resultset from
MySQL and then have a bunch of workes calling the command-line with the
arguments until there aren't anymore e-mail address. Can this be done in a
simple, yet stable, way?


Answer: you could use the multiprocessing module like this :

    
    
    from multiprocessing import Pool
    p = Pool()    # let python choose the optimal number of processes (= number of CPU cores)
    def treat_email(email_adress):
        # do the stuff you need with email address
    
    email_addresses = grab_the_list_from_mysql()  # something like "select mail from my_user_table"
    
    p.map(treat_email, email_addresses)    # this will treat all the emails in the X processes
    

