how to execute scrapy shell "URL" with notebook
Question: i am trying to scrapy

and there is scrapy shell "URL" command, executing console ipython with a
response object from the URL.

but i want to do that thing with notebook.

is there any way to execute scrapy shell with notebook,

or how can i get the same response object of the URL in ipython manually?


Answer: Stumbled across this unanswered question looking for the same answer. The
closest thing I can come up right now is this:

    
    
    import requests
    from scrapy.http import TextResponse
    
    r = requests.get('http://stackoverflow.com/')
    response = TextResponse(r.url, body=r.text, encoding='utf-8')
    

Then you can debug response using usual methods like xpath().

