How to crawl each and every link given on a website and collect all the text using scrapy
Question: I followed link

`https://stackoverflow.com/questions/19254630/how-to-use-scrapy-to-crawl-all-
items-in-a-website`

but things does not work out for me.

I am trying to learn scraping data over web.I was implementing tutorial given
on <http://scrapy.readthedocs.io/en/latest/intro/examples.html>

and able to crawl over a given link here is sample code snap

    
    
    from scrapy.spiders import Spider
    from scrapy.selector import Selector
    
    from dirbot.items import Website
    
    
    class DmozSpider(Spider):
        name = "dmoz"
        allowed_domains = ["dmoz.org"]
        start_urls = [
            "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
            "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/",
        ]
    
        def parse(self, response):
            """
            The lines below is a spider contract. For more info see:
            http://doc.scrapy.org/en/latest/topics/contracts.html
    
            @url http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/
            @scrapes name
            """
            sel = Selector(response)
            sites = sel.xpath('//ul[@class="directory-url"]/li')
            items = []
    
            for site in sites:
                item = Website()
                item['name'] = site.xpath('a/text()').extract()
                item['url'] = site.xpath('a/@href').extract()
                item['description'] = site.xpath('text()').re('-\s[^\n]*\\r')
                items.append(item)
    
            return items
    

and Code snap for Item is

    
    
     from scrapy.item import Item, Field
    
    
    class Website(Item):
    
        name = Field()
        description = Field()
        url = Field() 
    

I am able to run crawler using `scrapy crawl dmoz`

but not able to achieve following thing

    
    
    1. All given link on website
    2. Didn't get all text from all possible link 
    3. Want to save them to a file 
    

Can some one guide me , for what changes need to do in my code so that i can
achieve my objective ?


Answer:   1. All given link on website

The response doesn't have `class="directory-url"` in it. You can get all given
links from the website using `sites = sel.xpath('//a/@href')` instead of that.
filter the needed urls

Or

If you can start from the main domain (`http://www.dmoz.org/`),like

`doc = html.fromstring(response.body)` `sites =
doc.xpath('.//section[@id="category-section"]//aside')`

`for site in sites:` `item = StackDemoItem()` `item['name'] =
site.xpath('.//div/h2/a/text()')` `item['url'] =
site.xpath('.//div/h2/a/@href')`

you have to append the domain name with `item['url']` to get the proper url.Do
the same for other pages respective of there url path.

  2. Didn't get all text from all possible link 

Most of the links does't have text along with it. So you have to strip the
contents from the url itself.else `text = sel.xpath('//a/text()')`

  3. Want to save them to a file 

You can simply save the contents using `scrapy crawl your_crawler_name -o
out.csv` , use json or txt instead of csv for that kind of file.

