Not able to go on with the Request object and callback mechanism of scrapy spider-crawler
Question: I am new to Scrapy and python.This is my spider-crawler

    
    
    from scrapy.spider import Spider
    from scrapy.selector import Selector
    from scrapy.http import Request
    from tutorial.settings import *
    from tutorial.items import *
    
    class DmozSpider(Spider):
    
       name = "dmoz"
       allowed_domains = ["m.timesofindia.com"]
       start_urls =    ["http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html"]
    
        def parse(self, response):
            sel = Selector(response)
            torrent = DmozItem()
            items=[]
            links = sel.xpath('//div[@class="gapleftm"]/ul[@class="content"]/li')
            for ti in sel.xpath("//a[@class='pda']/text()").extract():
                yield DmozItem(title=ti)
            for url in sel.xpath("//a[@class='pda']/@href").extract():
                yield DmozItem(link=url)
                yield Request(url, callback=self.my_parse)
    
        def my_parse(self, response):
            sel = Selector(response)
            self.log('A response from my_parse just arrived!')
            for text in sel.xpath("//body/text()").extract():
                yield DmozItem(desc=text)
                pass
    

here i am trying to collect all the urls that are in tag and then calling my
callback function but code fails to enter my_parse function. Am I missing
something.

This is my console log

    
    
    root@yogesh-System-model:~/pythonTest/tutorial# scrapy crawl  dmoz -o mypune13.txt
    2014-02-06 16:15:01+0530 [scrapy] INFO: Scrapy 0.22.0 started (bot: tutorial)
    2014-02-06 16:15:01+0530 [scrapy] INFO: Optional features available: ssl, http11, boto,     django
    2014-02-06 16:15:01+0530 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE':    'tutorial.spiders', 'SPIDER_MODULES': ['tutorial.spiders'], 'FEED_URI': 'mypune13.txt',   'BOT_NAME': 'tutorial'}
    2014-02-06 16:15:01+0530 [scrapy] INFO: Enabled extensions: FeedExporter, LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState
    2014-02-06 16:15:02+0530 [scrapy] INFO: Enabled downloader middlewares:  HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware,   DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
    2014-02-06 16:15:02+0530 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
    2014-02-06 16:15:02+0530 [scrapy] INFO: Enabled item pipelines: 
    2014-02-06 16:15:02+0530 [dmoz] INFO: Spider opened
    2014-02-06 16:15:02+0530 [dmoz] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
    2014-02-06 16:15:02+0530 [scrapy] DEBUG: Telnet console listening on 0.0.0.0:6023
    2014-02-06 16:15:02+0530 [scrapy] DEBUG: Web service listening on 0.0.0.0:6080
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Crawled (200) <GET http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html> (referer: None)
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'title': u'Front Page'}
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'title': u'Times City'}
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'title': u'Times Nation'}
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'title': u'Auto Expo 2014'}
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'title': u'Times Global'}
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'title': u'Editorial'}
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'title': u'Times Business'}
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'title': u'Times Sport'}
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'title': u'Pune Times'}
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'title': u'NEWS DIGEST'}
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'title': u'Cong denied Pranab chance to be PM: Modi'}
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'title': u'Mom, daughter badly hurt in mishap at theme park'}
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'title': u'13 Indians now head major global firms,4 studied at St Stephens'}
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'title': u'9.7cr new voters added across India in 5 years'}
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'title': u'Exit bond money for AFMC grads hiked up to Rs 30 lakh'}
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'title': u'SC revisiting death sentences, stays 3 more'}
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'title': u'Jr college teachers call off HSC exams boycott plan'}
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'title': u'Tourists from 180 countries to get visa on arrival now'}
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'title': u'50 of 58 new Rajya Sabha members are crorepatis'}
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'title': u'2G spectrum bids touch Rs 50,000 crore'}
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'title': u'Discoms loss may be Tata Powers gain'}
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'title': u'Colleges, schools work till last min to give hall tickets'}
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'title': u'Front Page'}
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'title': u'Times City'}
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'title': u'Times Nation'}
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'title': u'Auto Expo 2014'}
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'title': u'Times Global'}
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'title': u'Editorial'}
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'title': u'Times Business'}
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'title': u'Times Sport'}
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'title': u'Pune Times'}
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'link': u'http://mobiletoi.timesofindia.com/mobile.aspx?sect_articles=yes&sectname=Front+Page&edid=&edlabel=TOIPU&mydateHid=06-02-2014&pubname=Times+of+India+-+Pune+-+Front+Page&edname=&publabel=TOI'}
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Filtered offsite request to 'mobiletoi.timesofindia.com': <GET http://mobiletoi.timesofindia.com/mobile.aspx?sect_articles=yes&sectname=Front+Page&edid=&edlabel=TOIPU&mydateHid=06-02-2014&pubname=Times+of+India+-+Pune+-+Front+Page&edname=&publabel=TOI>
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'link': u'http://mobiletoi.timesofindia.com/mobile.aspx?sect_articles=yes&sectname=Times+City&edid=&edlabel=TOIPU&mydateHid=06-02-2014&pubname=Times+of+India+-+Pune+-+Times+City&edname=&publabel=TOI'}
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'link': u'http://mobiletoi.timesofindia.com/mobile.aspx?sect_articles=yes&sectname=Times+Nation&edid=&edlabel=TOIPU&mydateHid=06-02-2014&pubname=Times+of+India+-+Pune+-+Times+Nation&edname=&publabel=TOI'}
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'link': u'http://mobiletoi.timesofindia.com/mobile.aspx?sect_articles=yes&sectname=Auto+Expo+2014&edid=&edlabel=TOIPU&mydateHid=06-02-2014&pubname=Times+of+India+-+Pune+-+Auto+Expo+2014&edname=&publabel=TOI'}
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'link': u'http://mobiletoi.timesofindia.com/mobile.aspx?sect_articles=yes&sectname=Times+Global&edid=&edlabel=TOIPU&mydateHid=06-02-2014&pubname=Times+of+India+-+Pune+-+Times+Global&edname=&publabel=TOI'}
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'link': u'http://mobiletoi.timesofindia.com/mobile.aspx?sect_articles=yes&sectname=Editorial&edid=&edlabel=TOIPU&mydateHid=06-02-2014&pubname=Times+of+India+-+Pune+-+Editorial&edname=&publabel=TOI'}
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'link': u'http://mobiletoi.timesofindia.com/mobile.aspx?sect_articles=yes&sectname=Times+Business&edid=&edlabel=TOIPU&mydateHid=06-02-2014&pubname=Times+of+India+-+Pune+-+Times+Business&edname=&publabel=TOI'}
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'link': u'http://mobiletoi.timesofindia.com/mobile.aspx?sect_articles=yes&sectname=Times+Sport&edid=&edlabel=TOIPU&mydateHid=06-02-2014&pubname=Times+of+India+-+Pune+-+Times+Sport&edname=&publabel=TOI'}
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'link': u'http://mobiletoi.timesofindia.com/mobile.aspx?sect_articles=yes&sectname=Pune+Times&edid=&edlabel=TOIPU&mydateHid=06-02-2014&pubname=Times+of+India+-+Pune+-+Pune+Times&edname=&publabel=TOI'}
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'link': u'http://mobiletoi.timesofindia.com/mobile.aspx?article=yes&pageid=3&sectid=edid=&edlabel=TOIPU&mydateHid=06-02-2014&pubname=Times+of+India+-+Pune&edname=&articleid=Ar00300&publabel=TOI'}
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'link': u'http://mobiletoi.timesofindia.com/mobile.aspx?article=yes&pageid=3&sectid=edid=&edlabel=TOIPU&mydateHid=06-02-2014&pubname=Times+of+India+-+Pune&edname=&articleid=Ar00301&publabel=TOI'}
    2014-02-06 16:15:03+0530 [dmoz] DEBUG: Scraped from <200 http://mobiletoi.timesofindia.com/htmldbtoi/TOIPU/20140206/TOIPU_articles__20140206.html>
    {'link': u'http://mobiletoi.timesofindia.com/mobile.aspx?article=yes&pageid=3&sectid=edid=&edlabel=TOIPU&mydateHid=06-02-2014&pubname=Times+of+India+-+Pune&edname=&articleid=Ar00302&publabel=TOI'}    
    2014-02-06 16:15:03+0530 [dmoz] INFO: Closing spider (finished)
    2014-02-06 16:15:03+0530 [dmoz] INFO: Stored jsonlines feed (62 items) in: mypune13.txt
    2014-02-06 16:15:03+0530 [dmoz] INFO: Dumping Scrapy stats:
        {'downloader/request_bytes': 279,
         'downloader/request_count': 1,
         'downloader/request_method_count/GET': 1,
         'downloader/response_bytes': 11226,
         'downloader/response_count': 1,
         'downloader/response_status_count/200': 1,
     'finish_reason': 'finished',
     'finish_time': datetime.datetime(2014, 2, 6, 10, 45, 3, 542688),
     'item_scraped_count': 62,
     'log_count/DEBUG': 66,
     'log_count/INFO': 8,
     'request_depth_max': 1,
     'response_received_count': 1,
     'scheduler/dequeued': 1,
     'scheduler/dequeued/memory': 1,
     'scheduler/enqueued': 1,
     'scheduler/enqueued/memory': 1,
     'start_time': datetime.datetime(2014, 2, 6, 10, 45, 2, 127946)}
    2014-02-06 16:15:03+0530 [dmoz] INFO: Spider closed (finished)
    


Answer: Your console log shows that your request for
`http://mobiletoi.timesofindia.com/mobile.aspx?sect_articles=yes&sectname=Front+Page&edid=&edlabel=TOIPU&mydateHid=06-02-2014&pubname=Times+of+India+-+Pune+-+Front+Page&edname=&publabel=TOI`
was filtered

    
    
    Filtered offsite request to 'mobiletoi.timesofindia.com'
    

Scrapy has an
[`OffsiteMiddleware`](http://doc.scrapy.org/en/latest/topics/spider-
middleware.html#scrapy.contrib.spidermiddleware.offsite.OffsiteMiddleware) on
by default:

> This middleware filters out every request whose host names aren’t in the
> spider’s allowed_domains attribute.

You need to include 'mobiletoi.timesofindia.com' in `allowed_domains`, like
this:

    
    
    allowed_domains = ["m.timesofindia.com", "mobiletoi.timesofindia.com"]
    

Otherwise, Scrapy spider middleware `OffsiteMiddleware` will receive your
requests that were yield with `yield Request(url, callback=self.my_parse)` and
say that domain doesn't match, and will discard them, with no callback being
called at all.

