title: Scrapy Shell Unresponsive for Certain Urls
body: <p>I'm working on a web crawler on the NIH grant database and I am running into an issue with scrapy's shell when I try to inspect the response object. </p>
  
  <p>After entering this command:</p>
  
  <pre><code>scrapy shell https://projectreporter.nih.gov/project_info_results.cfm?aid=8535284&amp;icde=27160266
  </code></pre>
  
  <p>I get this output as the shell starts up:</p>
  
  <pre><code>$ 2015-11-20 01:18:10 [scrapy] INFO: 
  
  Scrapy 1.0.3 started (bot: tutorial)
  2015-11-20 01:18:10 [scrapy] INFO: Optional features available: ssl, http11
  2015-11-20 01:18:10 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'tutorial.spiders', 'SPIDER_MODULES': ['tutorial.spiders'], 'LOGSTATS_INTERVAL': 0, 'BOT_NAME': 'tutorial'}
  2015-11-20 01:18:10 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, CoreStats, SpiderState
  2015-11-20 01:18:10 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
  2015-11-20 01:18:10 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
  2015-11-20 01:18:10 [scrapy] INFO: Enabled item pipelines: 
  2015-11-20 01:18:10 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6027
  2015-11-20 01:18:10 [scrapy] INFO: Spider opened
  2015-11-20 01:18:11 [scrapy] DEBUG: Crawled (200) &lt;GET https://projectreporter.nih.gov/project_info_results.cfm?aid=8535284&gt; (referer: None)
  [s] Available Scrapy objects:
  [s]   crawler    &lt;scrapy.crawler.Crawler object at 0x101465d10&gt;
  [s]   item       {}
  [s]   request    &lt;GET https://projectreporter.nih.gov/project_info_results.cfm?aid=8535284&gt;
  [s]   response   &lt;200 https://projectreporter.nih.gov/project_info_results.cfm?aid=8535284&gt;
  [s]   settings   &lt;scrapy.settings.Settings object at 0x1028bbbd0&gt;
  [s]   spider     &lt;DefaultSpider 'default' at 0x1047b7510&gt;
  [s] Useful shortcuts:
  [s]   shelp()           Shell help (print this help)
  [s]   fetch(req_or_url) Fetch request (or URL) and update local objects
  [s]   view(response)    View response in a browser
  2015-11-20 01:18:12 [root] DEBUG: Using default logger
  2015-11-20 01:18:12 [root] DEBUG: Using default logger
  </code></pre>
  
  <p>Unfortunately it freezes there and never continues on, therefore I can't manipulate the response object or move forward. </p>
  
  <p>If I put in the same command with any other url, I get this response almost immediately:</p>
  
  <pre><code>$ scrapy shell https://reddit.com
  2015-11-20 01:20:24 [scrapy] INFO: Scrapy 1.0.3 started (bot: tutorial)
  2015-11-20 01:20:24 [scrapy] INFO: Optional features available: ssl, http11
  2015-11-20 01:20:24 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'tutorial.spiders', 'SPIDER_MODULES': ['tutorial.spiders'], 'LOGSTATS_INTERVAL': 0, 'BOT_NAME': 'tutorial'}
  2015-11-20 01:20:24 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, CoreStats, SpiderState
  2015-11-20 01:20:24 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
  2015-11-20 01:20:24 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
  2015-11-20 01:20:24 [scrapy] INFO: Enabled item pipelines: 
  2015-11-20 01:20:24 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6028
  2015-11-20 01:20:24 [scrapy] INFO: Spider opened
  2015-11-20 01:20:24 [scrapy] DEBUG: Redirecting (301) to &lt;GET https://www.reddit.com/&gt; from &lt;GET https://reddit.com&gt;
  2015-11-20 01:20:25 [scrapy] DEBUG: Crawled (200) &lt;GET https://www.reddit.com/&gt; (referer: None)
  [s] Available Scrapy objects:
  [s]   crawler    &lt;scrapy.crawler.Crawler object at 0x107ea0d10&gt;
  [s]   item       {}
  [s]   request    &lt;GET https://reddit.com&gt;
  [s]   response   &lt;200 https://www.reddit.com/&gt;
  [s]   settings   &lt;scrapy.settings.Settings object at 0x1092f7bd0&gt;
  [s]   spider     &lt;DefaultSpider 'default' at 0x10ba31490&gt;
  [s] Useful shortcuts:
  [s]   shelp()           Shell help (print this help)
  [s]   fetch(req_or_url) Fetch request (or URL) and update local objects
  [s]   view(response)    View response in a browser
  2015-11-20 01:20:26 [root] DEBUG: Using default logger
  2015-11-20 01:20:26 [root] DEBUG: Using default logger
  
  In [1]: 
  </code></pre>
  
  <p>^^^ the 'In [1]:' indicating the request and response was carried out correctly. </p>
  
  <p>What's more, if I download the html of my initial url as a local file via 'Save As' and query that using the scrapy command:</p>
  
  <pre><code>$ scrapy shell file://127.0.0.1/path/to/file.html 
  </code></pre>
  
  <p>I get the scrapy shell to work as intended.</p>
  
  <p>I guess this was a long winded way of asking, what is it about the url that I am using that is tripping up the scrapy shell? is it the fact that I had to submit a search to arrive at the page? something else? Any help would be greatly appreciated.</p>