I am currently using scrapy's CrawlSpider to look for specific info on a list of multiple start\_urls. What I would like to do is stop scraping a specific start\_url's domain once I've found the information I've looked for, so it won't keep hitting a domain and will instead just hit the other start\_urls.

Is there a way to do this? I have tried appending to deny\_domains like so:

```
deniedDomains = []
...
rules = [Rule(SgmlLinkExtractor(..., deny_domains=(etc), ...)]
...
def parseURL(self, response):
    ...
    self.deniedDomains.append(specificDomain)

```

Appending doesn't seem to stop the crawling, but if I start the spider with the intended specificDomain then it'll stop as requested. So I'm assuming that you can't change the deny\_domains list after the spider's started?