How to read xml directly from URLs with scrapy/python
Question: In Scrapy you will have to define `start_url`s. But how can I crawl from other
urls as well?

Up to now I have a login script which logs into a webpage. After logging in, I
want to extract xml from different urls.

    
    
    import scrapy
    
    class LoginSpider(scrapy.Spider):
        name = 'example'
        start_urls = ['login page']
        urls = ['url','url']
    
    def parse(self, response):
        return scrapy.FormRequest.from_response(
            response,
            formdata={'UserName': '', 'Password': ''},
            callback=self.check_login_response
        )
    
    def check_login_response(self, response):
        # check login succeed before going on
        if "incorrect" in response.body:
            self.log("Login failed", level=scrapy.log.ERROR)
            return
    

How is it possible to crawl data from the urls defined in the urls array?


Answer: You need to `yield`
[`Request`](http://doc.scrapy.org/en/latest/topics/request-
response.html#scrapy.http.Request) instances to the other URLs:

    
    
    def check_login_response(self, response):
        # check login succeed before going on
        if "incorrect" in response.body:
            self.log("Login failed", level=scrapy.log.ERROR)
            return
    
        for url in list_or_urls:
            yield Request(url, callback=self.parse_other_url)
    
    def parse_other_url(self, response):
        # ...
    

