How would you make this script run faster?
Question: 
    import urllib2
    import time
    
    def hunt(url, start="<blockquote>", end="</blockquote>"):
        while 1:
            x = urllib2.urlopen(url)
            y = x.read()
            print y[y.find(start):y.find(end)]
            time.sleep(1)
    

I'm trying to get constant updates of a single element on a web page,
including a time interval to avoid getting banned by the server. It doesn't
have to be python, btw.


Answer: Let's try an experiment to compare the speed of `str.find()` to `re.search()`:

    
    
    import timeit
    
    setup = '''
    import urllib2
    import re
    start = "<body>"
    end = "</body>"
    url = 'http://www.stackoverflow.com'
    req = urllib2.urlopen(url)
    res = req.read()
    regex = re.compile('%s.+?%s' % (start, end))
    '''
    
    timeit.timeit('''res[res.find(start):res.find(end)]''',
        setup = setup, number = 1000)
    
    timeit.timeit('''res[res.find(start):res.rfind(end)]''',
        setup = setup, number = 1000)
    
    timeit.timeit('''regex.search(res)''',
        setup = setup, number = 1000)
    

With this we get:

0.16357661195633

0.08454644330907968

0.2768974693601649

So it looks like `str.find()` has a pretty decent speed, but if you know your
end quote is going to be closer to the end than the beginning you can speed
things up with `str.rfind()`.

Another thing you can do is to use multiple threads. Start one thread that
constantly fetches URLs and puts them in a Queue, then have another thread
process the Queue. This way, when the first thread sleeps while waiting on IO,
the second thread will be processing the string from the previous URL.
Something roughly along these lines:

    
    
    import Queue
    import threading
    import urllib2
    
    q = Queue.Queue()
    results = []
    
    url = 'http://www.google.com/'
    start = '<body>'
    end = '</body>'
    
    def get_urls():
        while 1:
            req = urllib2.urlopen(url)
            res = req.read()
            print "putting data len", len(res)
            q.put(res)
    
    def process_url():
        url_data = q.get()
        result = url_data[url_data.find(start):url_data.find(end)]
        results.append(result)
        q.task_done()
    
    putter_thread = threading.Thread(target = get_urls)
    getter_thread = threading.Thread(target = process_url)
    
    putter_thread.start()
    getter_thread.start()
    

