How do I use the Python Scrapy module to list all the URLs from my website?
Question: I want to use the Python [Scrapy module](http://scrapy.org/) to scrape all the
URLs from my website and write the list to a file. I looked in the examples
but didn't see any simple example to do this.


Answer: Here's the python program that worked for me:

    
    
    from scrapy.selector import HtmlXPathSelector
    from scrapy.spider import BaseSpider
    from scrapy.http import Request
    
    DOMAIN = 'example.com'
    URL = 'http://%s' % DOMAIN
    
    class MySpider(BaseSpider):
        name = DOMAIN
        allowed_domains = [DOMAIN]
        start_urls = [
            URL
        ]
    
        def parse(self, response):
            hxs = HtmlXPathSelector(response)
            for url in hxs.select('//a/@href').extract():
                if not ( url.startswith('http://') or url.startswith('https://') ):
                    url= URL + url 
                print url
                yield Request(url, callback=self.parse)
    

Save this in a file called `spider.py`.

You can then use a shell pipeline to post process this text:

    
    
    bash$ scrapy runspider spider.py > urls.out
    bash$ cat urls.out| grep 'example.com' |sort |uniq |grep -v '#' |grep -v 'mailto' > example.urls
    

This gives me a list of all the unique urls in my site.

