get many pages with pycurl?
Question: I want to get many pages from a website, like

    
    
    curl "http://farmsubsidy.org/DE/browse?page=[0000-3603]" -o "de.#1"
    

but get the pages' data in python, not disk files. Can someone please post
`pycurl` code to do this,  
or fast `urllib2` (not one-at-a-time) if that's possible,  
or else say "forget it, curl is faster and more robust" ? Thanks


Answer: So you have 2 problem and let me show you in one example. Notice the pycurl
already did the multithreading/not one-at-a-time w/o your hardwork.

    
    
    #! /usr/bin/env python
    
    import sys, select, time
    import pycurl,StringIO
    
    c1 = pycurl.Curl()
    c2 = pycurl.Curl()
    c3 = pycurl.Curl()
    c1.setopt(c1.URL, "http://www.python.org")
    c2.setopt(c2.URL, "http://curl.haxx.se")
    c3.setopt(c3.URL, "http://slashdot.org")
    s1 = StringIO.StringIO()
    s2 = StringIO.StringIO()
    s3 = StringIO.StringIO()
    c1.setopt(c1.WRITEFUNCTION, s1.write)
    c2.setopt(c2.WRITEFUNCTION, s2.write)
    c3.setopt(c3.WRITEFUNCTION, s3.write)
    
    m = pycurl.CurlMulti()
    m.add_handle(c1)
    m.add_handle(c2)
    m.add_handle(c3)
    
    # Number of seconds to wait for a timeout to happen
    SELECT_TIMEOUT = 1.0
    
    # Stir the state machine into action
    while 1:
        ret, num_handles = m.perform()
        if ret != pycurl.E_CALL_MULTI_PERFORM:
            break
    
    # Keep going until all the connections have terminated
    while num_handles:
        # The select method uses fdset internally to determine which file descriptors
        # to check.
        m.select(SELECT_TIMEOUT)
        while 1:
            ret, num_handles = m.perform()
            if ret != pycurl.E_CALL_MULTI_PERFORM:
                break
    
    # Cleanup
    m.remove_handle(c3)
    m.remove_handle(c2)
    m.remove_handle(c1)
    m.close()
    c1.close()
    c2.close()
    c3.close()
    print "http://www.python.org is ",s1.getvalue()
    print "http://curl.haxx.se is ",s2.getvalue()
    print "http://slashdot.org is ",s3.getvalue()
    

Finally, these code is mainly based on an example on the pycurl site =.=

may be you should really read doc. ppl spend huge time on it.

