Python - Limiting Number of Threads while Multithreading
Question: Code below.

    
    
                import sys
                import urllib2
                import threading
                import time
    
                urls = ["http://www.google.com", "http://www.apple.com"]
    
                def fetch_url(url):
                    html = urllib2.urlopen(url).read()
                    print html
    
                f = open("Output.txt", "w")
                e = open("ErrorUsers.txt", "w")
    
                threads = [threading.Thread(target=fetch_url, args=(url,)) for url in urls]
                for thread in threads:
                    try:
                        thread.daemon = True
                        thread.start()
                    except urllib2.HTTPError, e:
                        e.write(url + "\n")
                    except urllib2.URLError, e:
                        e.write(url + "\n")
                for thread in threads:
                    thread.join()
    

Essentially I need to grab data from a large number of sites (we're talking in
the tens of thousands). I'm using the above code as a base, which works great,
however, the code freezes after the creation of 750 threads of so. I'm
wondering how to limit the number of active threads, or to like close a thread
once it finishes.


Answer: You can use a thread pool like the one implemented in multiprocessing.In the
following snippet, a maximum of 100 threads will be active any time.

    
    
    from multiprocessing.pool import ThreadPool
    
    urls = ["http://www.google.com", "http://www.apple.com"]
    
    def fetch_url(url):
        html = urllib2.urlopen(url).read()
        print html
    
    pool = ThreadPool(100)
    pool.map(fetch_url, urls)
    pool.close()
    pool.join()
    

