<p dir="auto">run the spider with <code class="notranslate">scrapy runspider</code>, it also happens on <code class="notranslate">parse</code> and <code class="notranslate">crawl</code>.<br>
<a href="https://gist.github.com/nramirezuy/65faa56c7eab1e117d77">https://gist.github.com/nramirezuy/65faa56c7eab1e117d77</a></p>
<p dir="auto">Programming a <code class="notranslate">Crawler</code> from ipython notebook is causing <code class="notranslate">boto</code> to raise an error</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="ERROR:boto:Caught exception reading instance data
Traceback (most recent call last):
  File &quot;/usr/lib/python2.7/dist-packages/boto/utils.py&quot;, line 219, in retry_url
    r = opener.open(req)
  File &quot;/usr/lib/python2.7/urllib2.py&quot;, line 404, in open
    response = self._open(req, data)
  File &quot;/usr/lib/python2.7/urllib2.py&quot;, line 422, in _open
    '_open', req)
  File &quot;/usr/lib/python2.7/urllib2.py&quot;, line 382, in _call_chain
    result = func(*args)
  File &quot;/usr/lib/python2.7/urllib2.py&quot;, line 1214, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File &quot;/usr/lib/python2.7/urllib2.py&quot;, line 1184, in do_open
    raise URLError(err)
URLError: &lt;urlopen error timed out&gt;
ERROR:boto:Unable to read instance data, giving up
2015-03-24 00:58:54-0300 [fipe-periodo] INFO: Closing spider (finished)
2015-03-24 00:58:54-0300 [fipe-periodo] INFO: Dumping Scrapy stats:
        {'downloader/request_bytes': 308,
         'downloader/request_count': 1,
         'downloader/request_method_count/GET': 1,
         'downloader/response_bytes': 124031,
         'downloader/response_count': 1,
         'downloader/response_status_count/200': 1,
         'finish_reason': 'finished',
         'finish_time': datetime.datetime(2015, 3, 24, 3, 58, 54, 646566),
         'item_scraped_count': 171,
         'response_received_count': 1,
         'scheduler/dequeued': 1,
         'scheduler/dequeued/memory': 1,
         'scheduler/enqueued': 1,
         'scheduler/enqueued/memory': 1,
         'start_time': datetime.datetime(2015, 3, 24, 3, 58, 53, 978029)}
2015-03-24 00:58:54-0300 [fipe-periodo] INFO: Spider closed (finished)"><pre class="notranslate"><code class="notranslate">ERROR:boto:Caught exception reading instance data
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/boto/utils.py", line 219, in retry_url
    r = opener.open(req)
  File "/usr/lib/python2.7/urllib2.py", line 404, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 422, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1214, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1184, in do_open
    raise URLError(err)
URLError: &lt;urlopen error timed out&gt;
ERROR:boto:Unable to read instance data, giving up
2015-03-24 00:58:54-0300 [fipe-periodo] INFO: Closing spider (finished)
2015-03-24 00:58:54-0300 [fipe-periodo] INFO: Dumping Scrapy stats:
        {'downloader/request_bytes': 308,
         'downloader/request_count': 1,
         'downloader/request_method_count/GET': 1,
         'downloader/response_bytes': 124031,
         'downloader/response_count': 1,
         'downloader/response_status_count/200': 1,
         'finish_reason': 'finished',
         'finish_time': datetime.datetime(2015, 3, 24, 3, 58, 54, 646566),
         'item_scraped_count': 171,
         'response_received_count': 1,
         'scheduler/dequeued': 1,
         'scheduler/dequeued/memory': 1,
         'scheduler/enqueued': 1,
         'scheduler/enqueued/memory': 1,
         'start_time': datetime.datetime(2015, 3, 24, 3, 58, 53, 978029)}
2015-03-24 00:58:54-0300 [fipe-periodo] INFO: Spider closed (finished)
</code></pre></div>
<p dir="auto">We solved the problem by removing boto from <code class="notranslate">scrapy.optional_packages</code></p>
<p dir="auto">Here is the code</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from twisted.internet import reactor
from scrapy.crawler import Crawler
from scrapy import log, signals
from scrapy.utils.project import get_project_settings

from fipe_spiders.spiders.automoveis import FipePeriodo

from scrapy import optional_features
# uncommented fixes my issue
# optional_features.remove('boto')

def setup_crawler(Spider):
    spider = Spider()
    settings = get_project_settings()

    crawler = Crawler(settings)
    crawler.signals.connect(reactor.stop, signal=signals.spider_closed)
    crawler.configure()
    crawler.crawl(spider)
    crawler.start()

setup_crawler(FipePeriodo)

log.start()
reactor.run()
"><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">twisted</span>.<span class="pl-s1">internet</span> <span class="pl-k">import</span> <span class="pl-s1">reactor</span>
<span class="pl-k">from</span> <span class="pl-s1">scrapy</span>.<span class="pl-s1">crawler</span> <span class="pl-k">import</span> <span class="pl-v">Crawler</span>
<span class="pl-k">from</span> <span class="pl-s1">scrapy</span> <span class="pl-k">import</span> <span class="pl-s1">log</span>, <span class="pl-s1">signals</span>
<span class="pl-k">from</span> <span class="pl-s1">scrapy</span>.<span class="pl-s1">utils</span>.<span class="pl-s1">project</span> <span class="pl-k">import</span> <span class="pl-s1">get_project_settings</span>

<span class="pl-k">from</span> <span class="pl-s1">fipe_spiders</span>.<span class="pl-s1">spiders</span>.<span class="pl-s1">automoveis</span> <span class="pl-k">import</span> <span class="pl-v">FipePeriodo</span>

<span class="pl-k">from</span> <span class="pl-s1">scrapy</span> <span class="pl-k">import</span> <span class="pl-s1">optional_features</span>
<span class="pl-c"># uncommented fixes my issue</span>
<span class="pl-c"># optional_features.remove('boto')</span>

<span class="pl-k">def</span> <span class="pl-en">setup_crawler</span>(<span class="pl-v">Spider</span>):
    <span class="pl-s1">spider</span> <span class="pl-c1">=</span> <span class="pl-v">Spider</span>()
    <span class="pl-s1">settings</span> <span class="pl-c1">=</span> <span class="pl-en">get_project_settings</span>()

    <span class="pl-s1">crawler</span> <span class="pl-c1">=</span> <span class="pl-v">Crawler</span>(<span class="pl-s1">settings</span>)
    <span class="pl-s1">crawler</span>.<span class="pl-s1">signals</span>.<span class="pl-en">connect</span>(<span class="pl-s1">reactor</span>.<span class="pl-s1">stop</span>, <span class="pl-s1">signal</span><span class="pl-c1">=</span><span class="pl-s1">signals</span>.<span class="pl-s1">spider_closed</span>)
    <span class="pl-s1">crawler</span>.<span class="pl-en">configure</span>()
    <span class="pl-s1">crawler</span>.<span class="pl-en">crawl</span>(<span class="pl-s1">spider</span>)
    <span class="pl-s1">crawler</span>.<span class="pl-en">start</span>()

<span class="pl-en">setup_crawler</span>(<span class="pl-v">FipePeriodo</span>)

<span class="pl-s1">log</span>.<span class="pl-en">start</span>()
<span class="pl-s1">reactor</span>.<span class="pl-en">run</span>()</pre></div>