How to check if webpage are alive with python with multiproccessing
Question: I have a list of urls (about 25k) and I am trying to check if they are alive
(200 response). want to do these checks in parallel using the multiprocessing
library for Python. I wrote the following (largely based on the Python doc
example) but it seems to run pretty slowly. Is there any way I can make this
script run faster?

    
    
        import urllib2
        import time
        import random
    
        from multiprocessing import Process, Queue, current_process, freeze_support
    
        class HeadRequest(urllib2.Request):
            def get_method(self):
                return "HEAD"
        #
        # Function run by worker processes
        #
    
        def worker(input, output):
            for args in iter(input.get, 'STOP'):
                result = alive(args) 
                output.put(result)
    
        #
        # Functions referenced by tasks
        #
    
        def alive(x):
            x = x.strip()
            try:
                return x, ":", urllib2.urlopen(HeadRequest(x)).getcode()
            except urllib2.HTTPError as e:
                return x, ":", e.code
            except:
                return x, ": Error"
    
        #
        #
        #
    
        def check():
            NUMBER_OF_PROCESSES = 500
            text_file = open("url.txt", "r")
            TASKS1 = text_file.readlines()
    
            # Create queues
            task_queue = Queue()
            done_queue = Queue()
    
            # Submit tasks
            for task in TASKS1:
                task_queue.put(task)
    
            # Start worker processes
            for i in range(NUMBER_OF_PROCESSES):
                Process(target=worker, args=(task_queue, done_queue)).start()
    
            # Get and print results
            for i in range(len(TASKS1)):
                print done_queue.get()
    
            # Tell child processes to stop
            for i in range(NUMBER_OF_PROCESSES):
                task_queue.put('STOP')
    
    
        if __name__ == '__main__':
            freeze_support()
            check()
    

Any help is appreciated


Answer: There is an easy way:

<http://scrapy.org/>

Scrapy provides web crawler framework for Python: you can give it a list of
URLs to crawl (in your case it doesn't need to follow links) and it will
automatically scale to several crawlers within the process/thread limitations
you give to it - you don't need to go to the details of multiprocess
communications and scaling yourself.

<http://doc.scrapy.org/topics/scrapyd.html#topics-scrapyd>

The only thing left for your own code is to analyze the results.

