python scrapy login redirecting problems
Question: I'm trying to use scrapy to crawl a website, but I'm not able to login to my
account through scrapy. Here is the spider code:

    
    
    from scrapy.spider import BaseSpider
    from scrapy.selector import HtmlXPathSelector
    from images.items import ImagesItem
    from scrapy.http import Request
    from scrapy.http import FormRequest
    from loginform import fill_login_form
    import requests
    import os
    import scrapy
    from scrapy.contrib.spiders import CrawlSpider, Rule
    from scrapy.shell import inspect_response
    
    class ImageSpider(BaseSpider):
       counter=0;
       name = "images"
       start_urls=['https://poshmark.com/login']
       f=open('poshmark.txt','wb')
       if not os.path.exists('./Image'):
          os.mkdir('./Image')
    
       def parse(self,response):
           return [FormRequest("https://www.poshmark.com/login",
                        formdata={
                             'login_form[username_email]':'Oliver1234',
                             'login_form[password]':'password'},
                        callback=self.real_parse)]
    
       def real_parse(self, response):
          print 'you are here'
          rq=[]
          mainsites=response.xpath("//body[@class='two-col feed one-col']/div[@class='body-con']/div[@class='main-con clear-fix']/div[@class='right-col']/div[@id='tiles']/div[@class='listing-con shopping-tile masonry-brick']/a/@href").extract()
          for mainsite in mainsites:
             r=Request(mainsite, callback=self.get_image)
             rq.append(r)
          return rq
       def get_image(self, response):
          req=[]
          sites=response.xpath("//body[@class='two-col small fixed']/div[@class='body-con']/div[@class='main-con']/div[@class='right-col']/div[@class='listing-wrapper']/div[@class='listing']/div[@class='img-con']/img/@src").extract()
          for site in sites:
             r = Request('http:'+site, callback=self.DownLload)
             req.append(r)
          return req
    
       def DownLload(self, response):
          str = response.url[0:-3];
          self.counter = self.counter+1
          str = str.split('/');
          print '----------------Image Get----------------',self.counter,str[-1],'jpg'
          imgfile = open('./Image/'+str[-1]+"jpg",'wb')
          imgfile.write(response.body)
          imgfile.close()
    

And I get the command window output like below:

    
    
    C:\Python27\Scripts\tutorial\images>scrapy crawl images
    C:\Python27\Scripts\tutorial\images\images\spiders\images_spider.py:14: ScrapyDe
    precationWarning: images.spiders.images_spider.ImageSpider inherits from depreca
    ted class scrapy.spider.BaseSpider, please inherit from scrapy.spider.Spider. (w
    arning only on first subclass, there may be others)
      class ImageSpider(BaseSpider):
    2015-06-09 23:43:29-0400 [scrapy] INFO: Scrapy 0.24.6 started (bot: images)
    2015-06-09 23:43:29-0400 [scrapy] INFO: Optional features available: ssl, http11
    
    2015-06-09 23:43:29-0400 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE'
    : 'images.spiders', 'SPIDER_MODULES': ['images.spiders'], 'BOT_NAME': 'images'}
    2015-06-09 23:43:29-0400 [scrapy] INFO: Enabled extensions: LogStats, TelnetCons
    ole, CloseSpider, WebService, CoreStats, SpiderState
    2015-06-09 23:43:30-0400 [scrapy] INFO: Enabled downloader middlewares: HttpAuth
    Middleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, Def
    aultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, Redirec
    tMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
    2015-06-09 23:43:30-0400 [scrapy] INFO: Enabled spider middlewares: HttpErrorMid
    dleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddlew
    are
    2015-06-09 23:43:30-0400 [scrapy] INFO: Enabled item pipelines:
    2015-06-09 23:43:30-0400 [images] INFO: Spider opened
    2015-06-09 23:43:30-0400 [images] INFO: Crawled 0 pages (at 0 pages/min), scrape
    d 0 items (at 0 items/min)
    2015-06-09 23:43:30-0400 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6
    023
    2015-06-09 23:43:30-0400 [scrapy] DEBUG: Web service listening on 127.0.0.1:6080
    
    2015-06-09 23:43:33-0400 [images] DEBUG: Crawled (200) <GET https://poshmark.com
    /login> (referer: None)
    2015-06-09 23:43:35-0400 [images] DEBUG: Redirecting (302) to <GET https://www.p
    oshmark.com/feed> from <POST https://www.poshmark.com/login>
    2015-06-09 23:43:35-0400 [images] DEBUG: Redirecting (301) to <GET https://poshm
    ark.com/feed> from <GET https://www.poshmark.com/feed>
    2015-06-09 23:43:36-0400 [images] DEBUG: Redirecting (302) to <GET https://poshm
    ark.com/login?pmrd%5Burl%5D=%2Ffeed> from <GET https://poshmark.com/feed>
    2015-06-09 23:43:36-0400 [images] DEBUG: Redirecting (301) to <GET https://poshm
    ark.com/login> from <GET https://poshmark.com/login?pmrd%5Burl%5D=%2Ffeed>
    2015-06-09 23:43:37-0400 [images] DEBUG: Crawled (200) <GET https://poshmark.com
    /login> (referer: https://poshmark.com/login)
    you are here
    2015-06-09 23:43:37-0400 [images] INFO: Closing spider (finished)
    2015-06-09 23:43:37-0400 [images] INFO: Dumping Scrapy stats:
            {'downloader/request_bytes': 4213,
             'downloader/request_count': 6,
             'downloader/request_method_count/GET': 5,
             'downloader/request_method_count/POST': 1,
             'downloader/response_bytes': 9535,
             'downloader/response_count': 6,
             'downloader/response_status_count/200': 2,
             'downloader/response_status_count/301': 2,
             'downloader/response_status_count/302': 2,
             'finish_reason': 'finished',
             'finish_time': datetime.datetime(2015, 6, 10, 3, 43, 37, 213000),
             'log_count/DEBUG': 8,
             'log_count/INFO': 7,
             'request_depth_max': 1,
             'response_received_count': 2,
             'scheduler/dequeued': 6,
             'scheduler/dequeued/memory': 6,
             'scheduler/enqueued': 6,
             'scheduler/enqueued/memory': 6,
             'start_time': datetime.datetime(2015, 6, 10, 3, 43, 30, 788000)}
    2015-06-09 23:43:37-0400 [images] INFO: Spider closed (finished)
    

You can see that it is redirected to ./feed from ./login, which seems like a
successful login, at the beginning, but then redirected back to ./login in the
end. Any ideas about what might be causing it?


Answer: When you log onto a website it stores (depending on the method of
authentication it could be different) some sort of token on the users session.
The problem you are having is that while you are getting authenticated
properly, your session data (the way the browser is able to tell the server
you are logged in and you are who you say you are) isn't being saved.

The person in this thread seems to have managed to do what you are seeking to
do here:

[Crawling with an authenticated session in
Scrapy](http://stackoverflow.com/questions/5851213/crawling-with-an-
authenticated-session-in-scrapy)

and here:

[Using Scrapy with authenticated (logged in) user
session](http://stackoverflow.com/questions/5850755/using-scrapy-with-
authenticated-logged-in-user-session)

