Cookie handling with Scrapy during login
Question: I'm trying to crawl some data from Amazon Mechanical Turkey, where I could
only view the first few pages of the result without logging in. It turned out
that Amazon requires cookies to record sessions, so the simplest way that just
submit a formrequest as many examples do won't work.

I've tried to pass some cookies around, though I thought scrapy will handle
that automatically, but it's not working. If I do open_in_browser after
submitting the form, I get the amazon page saying that I should enable cookies
in order to logging.

Then I came to another post where he uses selenium to get the cookies. I've
also tried it and the same occurs.

Here's what I've got right now. I've add `COOKIES_ENABLED = True` to
`settings.py`

By adding COOKIES_DEBUG to settings, I think cookies are received and set by
looking at the log simply with InitSpider, without selenium. But it just wont
work.

    
    
    from scrapy.spider import Spider
    from scrapy.selector import Selector
    from scrapy.http import Request, FormRequest
    from scrapy.contrib.linkextractors import LinkExtractor
    from scrapy.utils.response import open_in_browser
    from mturk.items import MturkItem
    from selenium import webdriver
    
    class MturkSpider(Spider):
        name = "AMT"
        allowed_domains = ["mturk.com","amazon.com"]
        start_url='https://www.mturk.com/mturk/viewhits?searchWords=&selectedSearchType=hitgroups&sortType=Title%3A1&pageNumber=1&searchSpec=HITGroupSearch%23T%231%2310%23-1%23T%23%21%23%21Title%211%21%23%21'
        login_page = "https://www.mturk.com/mturk/beginsignin"
        formdata ={'create':'0','email': 'xxx@example.com', 'password': '1234'}
    
    
        def get_cookies(self):
            driver = webdriver.Firefox()
            driver.implicitly_wait(30)
            base_url = "https://www.mturk.com/mturk/beginsignin"
            driver.get(base_url)
            driver.find_element_by_name("email").clear()
            driver.find_element_by_name("email").send_keys("xxx@example.com")
            driver.find_element_by_name("password").clear()
            driver.find_element_by_name("password").send_keys("1234")
            driver.find_element_by_id("signInSubmit-input").click()
            cookies = driver.get_cookies()
            driver.close()
            return cookies
    
        def start_requests(self):
            self.my_cookies =  self.get_cookies()
            yield Request(self.login_page, 
                cookies = self.get_cookies(),
                callback = self.login,
            )
    
        def login(self, response):
            yield FormRequest.from_response(response, 
                formdata = self.formdata,
                # cookies=self.my_cookies,
                callback = self.after_login,
            )
    
        def after_login(self,response):
            open_in_browser(response) # where it says I need to enable cookies
            yield Request(self.start_url, 
                # cookies=self.my_cookies,
                callback = self.parse_page,
            )
    
        def parse_page(self, response):
            # do the parsing, where I can successfully crawl the first few pages
    

I'm fairly new to python and this society. I have to say my knowledge is quite
limited in this area and I can only learn from others' work. Anybody has some
suggestions to make it work?

I've found solutions with urllib2/mechanize regarding amazon login, but no
solution with Request. I thought the case was similar here?

UPDATE: I've solved it myself. Seems no need to use selenium at all. The
problem lies in that I have to specify proper headers during submission of
requests. I manually set all headers grabbed from my browser and it worked.


Answer: I had the exact same problem, but rather than manually setting the Request
headers, I found that one of the header values was for the User Agent. Setting
this value in my project's setting file did the trick and I now do not have to
hard-code the header values:

USER_AGENT = 'Mozilla/5.0 (Windows NT 6.1; rv:32.0) Gecko/20100101
Firefox/32.0'

or whatever value for the browser you're using.

Now, the cookies are working properly for me--automatically.

