title: Access session cookie in scrapy spiders
body: <p>I am trying to access the session cookie within a spider.  I first login to a social network using in a spider:</p>
  
  <pre><code>    def parse(self, response):
  
          return [FormRequest.from_response(response,
                  formname='login_form',
                  formdata={'email': '...', 'pass':'...'},
                  callback=self.after_login)]
  </code></pre>
  
  <p>In <code>after_login</code>, I would like to access the session cookies, in order to pass them to another module (selenium here) to further process the page with an authentificated session.</p>
  
  <p>I would like something like that:</p>
  
  <pre><code>     def after_login(self, response):
  
          # process response
          .....
  
          # access the cookies of that session to access another URL in the
          # same domain with the autehnticated session.
          # Something like:
          session_cookies = XXX.get_session_cookies()
          data = another_function(url,cookies)
  </code></pre>
  
  <p>Unfortunately, <code>response.cookies</code>  does not return the session cookies.</p>
  
  <p>How can I get the session cookies ?  I was looking at the cookies middleware: <em>scrapy.contrib.downloadermiddleware.cookies</em> and <em>scrapy.http.cookies</em> but there doesn't seem to be any straightforward way to access the session cookies.</p>
  
  <p><strong>Some more details here bout my original question:</strong></p>
  
  <p>Unfortunately, I used your idea but I dind't see the cookies, although I know for sure that they exists since the <em>scrapy.contrib.downloadermiddleware.cookies</em> middleware does print out the cookies!  These are exactly the cookies that I want to grab.</p>
  
  <p>So here is what I am doing:</p>
  
  <p>The after_login(self,response) method receives the response variable after proper authentication, and then I access an URL with the session data:</p>
  
  <pre><code>  def after_login(self, response):
  
          # testing to see if I can get the session cookies
          cookieJar = response.meta.setdefault('cookie_jar', CookieJar())
          cookieJar.extract_cookies(response, response.request)
          cookies_test = cookieJar._cookies
          print "cookies - test:",cookies_test
  
          # URL access with authenticated session
          url = "http://site.org/?id=XXXX"     
          request = Request(url=url,callback=self.get_pict)   
          return [request] 
  </code></pre>
  
  <p>As the output below shows, there are indeed cookies, but I fail to capture them with cookieJar:</p>
  
  <pre><code>cookies - test: {}
  2012-01-02 22:44:39-0800 [myspider] DEBUG: Sending cookies to: &lt;GET http://www.facebook.com/profile.php?id=529907453&gt;
      Cookie: xxx=3..........; yyy=34.............; zzz=.................; uuu=44..........
  </code></pre>
  
  <p>So I would like to get a dictionary containing the keys xxx, yyy etc with the corresponding values. </p>
  
  <p>Thanks :)</p>