title: Google App Engine - Basic Cookie Handling with URLFetch
body: <p>Experts,</p>
  
  <p>It seems that retrieving even the most basic websites using google app engine can be quite challenging!</p>
  
  <p>In my case I would like to retrieve the website at this url:</p>
  
  <p><a href="http://tdbank.mortgagewebcenter.com/PowerSite/CheckRates.aspx/Index/9809" rel="nofollow">http://tdbank.mortgagewebcenter.com/PowerSite/CheckRates.aspx/Index/9809</a></p>
  
  <p>I would like to accept all of the cookies and then POST a response to this url (this is a simple form post):</p>
  
  <p><a href="http://tdbank.mortgagewebcenter.com/PowerSite/CheckRates.aspx/Search" rel="nofollow">http://tdbank.mortgagewebcenter.com/PowerSite/CheckRates.aspx/Search</a></p>
  
  <p>The string I would like to POST is:</p>
  
  <pre><code>'POSTDATA': 'Q585=1&amp;Q2926=1&amp;Q586=200000&amp;Q587=240000&amp;Q588=&amp;Q9166=07071&amp;Q591=1&amp;Q592=1&amp;Q594=3&amp;searchButton=Search'
  </code></pre>
  
  <p>The problem I am getting is that the webpage is stating that 'Cookies are not enabled' on my web browser.</p>
  
  <p>As you can see from the code below I tried manually adding cookies however this was not successful.</p>
  
  <p>Please help!
  -Todd</p>
  
  <pre><code>import cgi
  import webapp2
  import gzip
  import StringIO
  
  from google.appengine.api import users
  from google.appengine.api import urlfetch
  from BeautifulSoup import BeautifulSoup
  
  class MainPage(webapp2.RequestHandler):
      def get(self):
  #        self.response.headers['Content-Type'] = 'text/html'
          url = "http://tdbank.mortgagewebcenter.com/PowerSite/CheckRates.aspx/Index/9809"
          result = urlfetch.fetch(url,
                                  headers={'User-Agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 5_1_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9B206 Safari/7534.48.3',
                                                   'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
                                                   'Accept-Language': 'en-us',
                                                   'Accept-Encoding': 'gzip',
                                                   'Connection': 'keep-alive'})
  
          cookie = result.headers.get('set-cookie')
          input_text = {'User-Agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 5_1_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9B206 Safari/7534.48.3',
                        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
                        'Accept-Language': 'en-us',
                        'Accept-Encoding': 'gzip',
                        'Connection': 'keep-alive',
                        'Content-Type': 'application/x-www-form-urlencoded',
                        'Content-Length': '97',
                        'POSTDATA': 'Q585=1&amp;Q2926=1&amp;Q586=200000&amp;Q587=240000&amp;Q588=&amp;Q9166=07071&amp;Q591=1&amp;Q592=1&amp;Q594=3&amp;searchButton=Search',
                        'SiteProfile':'ProfileId=9809',
                        's_sess':'c_m=undefinedfeedity.comfeedity.com; s_sq=; s_cc=true;;',
                        'bhCookieSaveSess':'1',
                        'bhPrevResults':'bhjs=1&amp;bhrf=http://www.google.com/'}
  
          input_text['set-cookie'] = cookie
  ##        self.response.out.write(input_text)
  
          url2 = "http://tdbank.mortgagewebcenter.com/PowerSite/CheckRates.aspx/Search"
          result2 = urlfetch.fetch(url2, method='POST',headers=input_text)
  
  #        self.response.out.write(result.headers)
          f = StringIO.StringIO(result2.content)
          c = gzip.GzipFile(fileobj=f)
          content = c.read()
          self.response.out.write(content)
  #        self.response.out.write(result.content)
  
  
  app = webapp2.WSGIApplication([('/', MainPage)],
                                debug=True)
  </code></pre>
  
  <p>yaml file:</p>
  
  <pre><code>application: fimrates
  version: 1
  runtime: python27
  api_version: 1
  threadsafe: true
  
  handlers:
  - url: /.*
    script: fimrates.app
  </code></pre>