Twisted Python getPage
Question: I tried to get support on this but I am TOTALLY confused.

Here's my code:

    
    
    
    from twisted.internet import reactor
    from twisted.web.client import getPage
    from twisted.web.error import Error
    from twisted.internet.defer import DeferredList
    from sys import argv
    
    class GrabPage:
     def __init__(self, page):
      self.page = page
    
     def start(self, *args):
      if args == ():
       # We apparently don't need authentication for this
       d1 = getPage(self.page)
      else:
       if len(args) == 2:
        # We have our login information
        d1 = getPage(self.page, headers={"Authorization": " ".join(args)})
       else:
        raise Exception('Missing parameters')
    
      d1.addCallback(self.pageCallback)
      dl = DeferredList([d1])
      d1.addErrback(self.errorHandler)
      dl.addCallback(self.listCallback)
    
     def errorHandler(self,result):
      # Bad thingy!
      pass
    
     def pageCallback(self, result):
      return result
    
     def listCallback(self, result):
      print result
    
    a = GrabPage('http://www.google.com')
    data = a.start() # Not the HTML
    

I wish to get the HTML out which is given to pageCallback when start() is
called. This has been a pita for me. Ty! And sorry for my sucky coding.


Answer: You're missing the basics of how Twisted operates. It all revolves around the
`reactor`, which you're never even running. Think of the reactor like this:

![Reactor Loop](http://krondo.com/blog/wp-
content/uploads/2009/07/reactor-1.png)

Until you start the reactor, by setting up deferreds all you're doing is
chaining them with no events from which to fire.

I recommend you give the [Twisted Intro](http://krondo.com/blog/?page_id=1327)
by [Dave Peticolas](http://krondo.com/) a read. It's quick and it really gives
you all the missing information that the Twisted documentation doesn't.

Anyways, here is the most basic usage example of `getPage` as possible:

    
    
    from twisted.web.client import getPage
    from twisted.internet import reactor
    
    url = 'http://aol.com'
    
    def print_and_stop(output):
        print output
        if reactor.running:
           reactor.stop()
    
    if __name__ == '__main__':
        print 'fetching', url
        d = getPage(url)
        d.addCallback(print_and_stop)
        reactor.run()
    

Since `getPage` returns a deferred, I'm adding the callback `print_and_stop`
to the deferred chain. After that, I start the `reactor`. The reactor fires
`getPage`, which then fires `print_and_stop` which prints the data from
aol.com and then stops the reactor.

**Edit to show a working example of OP's code:**

    
    
    class GrabPage:
        def __init__(self, page):
            self.page = page
            ########### I added this:
            self.data = None
    
        def start(self, *args):
            if args == ():
                # We apparently don't need authentication for this
                d1 = getPage(self.page)
            else:
                if len(args) == 2:
                    # We have our login information
                    d1 = getPage(self.page, headers={"Authorization": " ".join(args)})
                else:
                    raise Exception('Missing parameters')
    
            d1.addCallback(self.pageCallback)
            dl = DeferredList([d1])
            d1.addErrback(self.errorHandler)
            dl.addCallback(self.listCallback)
    
        def errorHandler(self,result):
            # Bad thingy!
            pass
    
        def pageCallback(self, result):
            ########### I added this, to hold the data:
            self.data = result
            return result
    
        def listCallback(self, result):
            print result
            # Added for effect:
            if reactor.running:
                reactor.stop()
    
    a = GrabPage('http://google.com')
    ########### Just call it without assigning to data
    #data = a.start() # Not the HTML
    a.start()
    
    ########### I added this:
    if not reactor.running:
        reactor.run()
    
    ########### Reference the data attribute from the class
    data = a.data
    print '------REACTOR STOPPED------'
    print
    ########### First 100 characters of a.data:
    print '------a.data[:100]------'
    print data[:100] 
    

