Spider won't run after updating Scrapy
Question: As seems to frequently happen here, I am quite new to Python 2.7 and Scrapy.
Our project has us scraping website date, following some links and more
scraping, and so on. This was all working fine. Then I updated Scrapy.

Now when I launch my spider, I get the following message: [![enter image
description
here](http://i.stack.imgur.com/y8rPg.jpg)](http://i.stack.imgur.com/y8rPg.jpg)

This wasn't coming up anywhere previously (none of my prior error messages
looked anything like this). I am now running scrapy 1.1.0 on Python 2.7. And
none of the spiders that had previously worked on this project are working.

I can provide some example code if need be, but my (admittedly limited)
knowledge of Python suggests to me that its not even getting to my script
before bombing out.

**EDIT:** OK, so this code is supposed to start at the first authors page for
Deakin University academics on The Conversation, and go through and scrape how
many articles they have written and comments they have made.

    
    
    import scrapy
    from ltuconver.items import ConversationItem
    from ltuconver.items import WebsitesItem
    from ltuconver.items import PersonItem
    from scrapy import Spider
    from scrapy.selector import Selector
    from scrapy.http import Request
    import bs4
    
    class ConversationSpider(scrapy.Spider):
    name = "urls"
    allowed_domains = ["theconversation.com"]
    start_urls = [
        'http://theconversation.com/institutions/deakin-university/authors']
    
    
    #URL grabber
    def parse(self, response):
        requests = []
        people = Selector(response).xpath('///*[@id="experts"]/ul[*]/li[*]')
        for person in people:
            item = WebsitesItem()
            item['url'] = 'http://theconversation.com/'+str(person.xpath('a/@href').extract())[4:-2]
            self.logger.info('parseURL = %s',item['url'])
            requests.append(Request(url=item['url'], callback=self.parseMainPage))
    
        soup = bs4.BeautifulSoup(response.body, 'html.parser')
        try:          
            nexturl = 'https://theconversation.com'+soup.find('span',class_='next').find('a')['href']
            requests.append(Request(url=nexturl))
        except:
            pass
        return requests
    
    #go to URLs are grab the info
    def parseMainPage(self, response):
        person = Selector(response)
        item = PersonItem()
        item['name'] = str(person.xpath('//*[@id="outer"]/header/div/div[2]/h1/text()').extract())[3:-2]
        item['occupation'] = str(person.xpath('//*[@id="outer"]/div/div[1]/div[1]/text()').extract())[11:-15]
        item['art_count'] = int(str(person.xpath('//*[@id="outer"]/header/div/div[3]/a[1]/h2/text()').extract())[3:-3])
        item['com_count'] = int(str(person.xpath('//*[@id="outer"]/header/div/div[3]/a[2]/h2/text()').extract())[3:-3])
    

And in my Settings, I have:

    
    
    BOT_NAME = 'ltuconver'
    SPIDER_MODULES = ['ltuconver.spiders']
    NEWSPIDER_MODULE = 'ltuconver.spiders'
    
    DEPTH_LIMIT=1
    


Answer: Apparently my six.py file was corrupt (or something like that). After swapping
it out with the same file from a colleague, it started working again 8-\

