Extraxt & save all text() with Scrapy
Question: I'd like to extract the text from an HTML file using Python. I want
essentially the same output I would get if I copied the text from a browser
and pasted it into notepad. To solve this problem I need to use the framework.
As an example, take a page <https://en.wikipedia.org/wiki/Main_Page>, thus
extract 100 pages without leaving the domain en.wikipedia.org


Answer: Simple basic code sample doing what you need.

    
    
    from scrapy import Spider
    
    
    class Foo(Spider):
        # start urls executed at the beginning
        # with default callback "parse"
        start_urls = ["https://en.wikipedia.org/wiki/Main_Page"]
        name = "basic_spider"
    
        def parse(self, response):
            # use css or xpath selectors to extract text
            print(response.css("::text").extract())
    

Save above as spider.py and run it with

    
    
    scrapy runspider spider.py
    

Start with [Scrapy
tutorial](http://doc.scrapy.org/en/latest/intro/tutorial.html) if you feel
something is not clear or needs improving feel free to improve the docs, they
are hosted on github.

Of course you must learn [Python first](https://docs.python.org/2/), so start
with learning Python if you dont know it already.

