Does Scrapy crawl ALL links with Rules?
Question: Code source: <http://mherman.org/blog/2012/11/08/recursively-scraping-web-
pages-with-scrapy/#rules> Im new to python and scrapy. I searched for
recursive spider and found this.

I have a few questions:

How does the follow work? Does it just takes href links from a page and add it
in to the request queue?

Which part of the web page does scrapy crawl from?

Does the code below scrape ALL links from a webpage?

Lets say i want to crawl and download every file from this website
<http://downloads.trendnet.com/>

the way i would probably do it is to scrape every link on this website and
check URL's content header and download if it is a file. Is this feasible?

Sorry if it is a bad question....

    
    
    from scrapy.contrib.spiders import CrawlSpider, Rule
    from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
    from scrapy.selector import HtmlXPathSelector
    from craigslist_sample.items import CraigslistSampleItem
    
    class MySpider(CrawlSpider):
        name = "craigs"
        allowed_domains = ["sfbay.craigslist.org"]
        start_urls = ["http://sfbay.craigslist.org/search/npo"]
    
        rules = (
            Rule(SgmlLinkExtractor(allow=(), restrict_xpaths=('//a[@class="button next"]',)), callback="parse_items", follow= True),
        )
    
        def parse_items(self, response):
            hxs = HtmlXPathSelector(response)
            titles = hxs.xpath('//span[@class="pl"]')
            items = []
            for titles in titles:
                item = CraigslistSampleItem()
                item["title"] = titles.xpath("a/text()").extract()
                item["link"] = titles.xpath("a/@href").extract()
                items.append(item)
            return(items)
    


Answer: I think RTFM is really really applicable here, but to give you a short answer:

**With regards to the example given**

    
    
    rules = (
            Rule(SgmlLinkExtractor(allow=(), restrict_xpaths=('//a[@class="button next"]',)), callback="parse_items", follow= True),
        )
    

You asked what it crawls. It only crawls what you set up under rules. THat
means that your bot only crawls the next page each time. For each page it
finds, it does: callback = parse_items.

    
    
    def parse_items(self, response):
            hxs = HtmlXPathSelector(response)
            titles = hxs.xpath('//span[@class="pl"]')
            items = []
            for titles in titles:
                item = CraigslistSampleItem()
                item["title"] = titles.xpath("a/text()").extract()
                item["link"] = titles.xpath("a/@href").extract()
                items.append(item)
            return(items)
    

What parse_items does in this case, is check for entries in a list. You define
the list via the xpath (as you can see above with `titles =
hxs.xpath('//span[@class="pl"]')`). For each entry in the list (i.e. `for
titles in titles:`), it copies the text and the link into an item. It then
returns the items (aka it's done).

Parse_items is done for each page the crawler finds by following the next-
button.

Under settings, you can include `DEPTH_LIMIT=3`. In this case, your
crawlspider will only crawl 3 deep.

**With regards to the site you posted:**

No, you don't need a crawlspider, since there are no multiple pages. A normal
base-spider is sufficient. A crawlspider however can work and I'll show some
bits below. Set the rules to restrict_xpath('//a',) and it'll follow all links
on the page.

Make sure your item.py contains all the necessary items. For example, below it
refers to item["link"]. In your item.py, make sure an item called link (caps-
sensitive) is included, i.e. make sure the line -- link = Field() -- is there.

Under parse_items, do something like this:

    
    
    def parse_items(self, response):
            list = response.xpath('//a"')
            items = []
            for titles in list:
                item = [INSERT WHATEVER YOU CALLED YOUR ITEM]
                item["title"] = titles.xpath("/text()").extract()
                item["link"] = titles.xpath("/@href").extract()
                if ".pdf" in item["link"]:
                    SEE COMMENT BELOW
            return(items)
    

The last bit you need to do is check how the item-pipeline works. It uses
file_urls etc. in your item.

