Constructing a regular expression for url in start_urls list in scrapy framework python
Question: I am very new to scrapy and also i didn't used regular expressions before

The following is my `spider.py` code

    
    
    class ExampleSpider(BaseSpider):
       name = "test_code
       allowed_domains = ["www.example.com"]
       start_urls = [
           "http://www.example.com/bookstore/new/1?filter=bookstore",
           "http://www.example.com/bookstore/new/2?filter=bookstore",
           "http://www.example.com/bookstore/new/3?filter=bookstore",
       ]
    
       def parse(self, response):
           hxs = HtmlXPathSelector(response)
    

Now if we look at `start_urls` all the three urls are same except they differ
at integer value `2?, 3?` and so on i mean unlimited according to urls present
on the site , i now that we can use crawlspider and we can construct regular
expression for the URL like below,

    
    
        from scrapy.contrib.spiders import CrawlSpider, Rule
        from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
        import re
    
        class ExampleSpider(CrawlSpider):
            name = 'example.com'
            allowed_domains = ['example.com']
            start_urls = [
           "http://www.example.com/bookstore/new/1?filter=bookstore",
           "http://www.example.com/bookstore/new/2?filter=bookstore",
           "http://www.example.com/bookstore/new/3?filter=bookstore",
       ]
    
            rules = (
                Rule(SgmlLinkExtractor(allow=(........),))),
            ) 
    
       def parse(self, response):
           hxs = HtmlXPathSelector(response)
    

can u please guide me , that how can i construct a crawl spider Rule for the
above `start_url` list.


Answer: If i understand you correctly, you want a lot of start URL with a certain
pattern.

If so, you can override
[BaseSpider.start_requests](http://doc.scrapy.org/en/latest/topics/spiders.html#scrapy.spider.BaseSpider.start_requests)
method:

    
    
    class ExampleSpider(BaseSpider):
        name = "test_code"
        allowed_domains = ["www.example.com"]
    
        def start_requests(self):
            for i in xrange(1000):
                yield self.make_requests_from_url("http://www.example.com/bookstore/new/%d?filter=bookstore" % i)
    
        ...
    

