Limiting number of processes in multiprocessing python
Question: My requirement is to generate `hundreds of HTTP POST requests per second`. I
am doing it using `urllib2`.

    
    
    def send():
        req = urllib2.Request(url)
        req.add_data(data)
        response = urllib2.urlopen(req)
    
    while datetime.datetime.now() <= ftime:
        p=Process(target=send, args=[])
        p.start()
        time.sleep(0.001)
    

The problem is this code `sometimes for some iterations` throws either of
following exceptions:

    
    
    HTTP 503 Service Unavailable.
    URLError: <urlopen error [Errno -2] Name or service not known>
    

I have tried using `requests(HTTP for humans)` as well but I am having some
proxy issues with that module. Seems like `requests` is sending http packets
to proxy server even when target machine is within same LAN. I don't want
packets to go to proxy server.


Answer: The simplest way to limit number of concurrent connections is to use a thread
pool:

    
    
    #!/usr/bin/env python
    from itertools import izip, repeat
    from multiprocessing.dummy import Pool # use threads for I/O bound tasks
    from urllib2 import urlopen
    
    def fetch(url_data):
        try:
            return url_data[0], urlopen(*url_data).read(), None
        except EnvironmentError as e:
            return url_data[0], None, str(e)
    
    if __name__=="__main__":
        pool = Pool(20) # use 20 concurrent connections
        params = izip(urls, repeat(data)) # use the same data for all urls
        for url, content, error in pool.imap_unorderred(fetch, params):
            if error is None:
               print("done: %s: %d" % (url, len(content)))
            else:
               print("error: %s: %s" % (url, error))
    

`503 Service Unavailable` is a server error. It might fail to handle the load.

`Name or service not known` is a dns error. If you need make many requests;
install/enable a local caching dns server.

