scrapy general parse workflow
Question: I'm new to python and scrapy and wish to understand the methodology. I have
tried the official tutorial on scrapy and followed it but it is only a basic
example. My requirement described below is different and only a little more
complex.

There is a site which displays Items from a db.  
For each Item, I need to take attributes from each individual Item page and
the search results (listings) page. The search results page URL is in the
format:

    
    
        http://example.com/search?&start_index=0
    

Changing _start_index_ will change where the results start from. Only 10
records are displayed per results page.

Results are displayed in table cells in the format:

    
    
        link | Desc. | Status
    

I need to retrieve Desc. and Status attributes, then follow the link to a page
containing more details, which I will also retrieve for Item.  
I wish to retrieve a given number of records from any starting index. My
current method using scrapy is shown below (edited for brevity):

    
    
    import scrapy
    
    from scrapy.exceptions import CloseSpider
    from cbury_scrapy.items import MyItem
    
    class ExampleSpider(scrapy.Spider):
        name = "example"
        allowed_domains = ["example.com"]
        start_urls = [
            "http://example.com/cgi/search?&start_index=",
        ]
    
        url_index = 0
        URLS_PER_PAGE = 10
        records_remaining = 16
        crawl_done = False
    
        da = MyItem()        
    
        def parse(self, response):
            while self.crawl_done != True:
                url = "http://example.com/cgi/search?&start_index=" + str(self.url_index)
                yield scrapy.Request(url, callback=self.parse_results)
                self.url_index += self.URLS_PER_PAGE
    
    
        def parse_results(self, response):
            # Retrieve all table rows from results page
            for row in response.xpath('//table/tr[@class="datrack_resultrow_odd" or @class="datrack_resultrow_even"]'):
                # extract the Description and Status fields
    
                # extract the link to Item page
                url = r.xpath('//td[@class="datrack_danumber_cell"]//@href').extract_first()
                yield scrapy.Request(url, callback=self.parse_item)
    
                if self.records_remaining == 0:
                    self.crawl_done = True
                    raise CloseSpider('Finished scrape of requested number of records.')
    
                self.records_remaining -= 1
    
        def parse_item(self, response):
            # get fields from item page
            # ...   
            yield self.item
    

The code currently does not stop when _records_remaining_ reaches 0 and even
after throwing _CloseSpider_ exception so that is a bug.

I feel this stems from being wrong in how the parsing methods are arranged.
What would be the correct way to structure this in the "scrapy" way? Any help
is appreciated.


Answer: 
    def parse(self, response):
        list_of_indexes = response.xpath('place xpath here that leads to a list of urls for indexes')
        for indexes in list_of_indexes:    
            #maybe the urls are only tags ie. ['/extension/for/index1', '/extension/for/index2', etc...]
            index_urls = ['http://domain.com' + index for index in indexes]
            yield scrapy.Request(index_urls, callback = self.parse_indexes)
    
    def parse_index(self, response):
        da = MyItem()
        da['record_date'] = response.xpath('xpath_here')
        da['record_summary'] = response.xpath('xpath_here')
        da['additional_record_info'] = response.xpath('xpath_here')
        yield da
    

This example is over-simplified but I hope it helps.

You want to instantiate your item `da = MyItem()` within the parse itself.

To answer the larger question about parse flow I would start with URLs. Once
you find the XPaths for the indexes from the start_url you'll use

    
    
    scrapy.Requests(URL = index_url, callback =parse_indexes)
    

This will direct your spider to the the next parse method parse_indexes.

index_url will be drawn from an iteration through the necessary xpaths.

parse_indexes will be just like parse but will then draw out the info from
the_next_index_url

If this answer is going in the right direction I can post an example later.

