Scrapy reverses order of arguments in url Python
Question: I'm running a scraper to crawl from <http://www.johnlscott.com/agent-
search.aspx> to the office rosters.

The office roster addresses look like this: [http://www.johnlscott.com/agent-
search.aspx?p=agentResults.asp&OfficeID=8627](http://www.johnlscott.com/agent-
search.aspx?p=agentResults.asp&OfficeID=8627) \- but Scrapy crawls
[http://www.johnlscott.com/agent-
search.aspx?OfficeID=8627&p=agentResults.asp](http://www.johnlscott.com/agent-
search.aspx?OfficeID=8627&p=agentResults.asp) which is a dead page. **The two
parts after .aspx are swapped.**

I even went so far as to manually load every single address explicitly as
start_urls, and it still happens.

I'm using the most recent Scrapy on python-2.7, Windows 8.1

Code Sample:

    
    
    class JLSSpider(CrawlSpider):
    
        name = 'JLS'
        allowed_domains = ['johnlscott.com']
        # start_urls = ['http://www.johnlscott.com/agent-search.aspx']
    
        rules = (
            Rule(callback="parse_start_url", follow=True),)
    
        def start_requests(self):
            with open('hrefnums.csv', 'rbU') as ifile:
                read = csv.reader(ifile)
                for row in read:
                    for col in row:
                        # I have a csv of the office IDs: (Just letting it crawl through them creates the same issue)
                        yield self.make_requests_from_url("http://www.johnlscott.com/agent-search.aspx?p=agentResults.asp&OfficeID=%s" % col)
    
    
        def parse_start_url(self, response):
            items = []
            sel = Selector(response)
            sections = sel.xpath("//tr/td/table[@id='tbAgents']/tr")
            for section in sections:
                item = JLSItem()
                item['name'] = section.xpath("td[2]/text()")[0].extract().replace(u'\xa0', ' ').strip()         
                items.append(item)
            return(items)
    


Answer: You can prevent the swapping of url parts by using the option
`canonicalize=False` like in the code:

    
    
    import scrapy
    from scrapy.contrib.spiders import CrawlSpider, Rule
    from scrapy.contrib.linkextractors import LinkExtractor
    
    class JLSSpider(CrawlSpider):
    
        name = 'JLS'
        allowed_domains = ['johnlscott.com']
        start_urls = ['http://www.johnlscott.com/agent-search.aspx']
    
        rules = (
            # http://www.johnlscott.com/agent-search.aspx?p=agentResults.asp&OfficeID=7859
            Rule(
                LinkExtractor(
                    allow=('p=agentResults.asp&OfficeID=', 
                    ), 
                    canonicalize=False
                ),
                callback='parse_roster',
                follow=True),
        )
    
        def parse_roster(self, response):
            pass
    

