How can I get all the plain text from a website with Scrapy?
Question: I would like to have all the text visible from a website, after the HTML is
rendered. I'm working in Python with Scrapy framework. With
`xpath('//body//text()')` I'm able to get it, but with the HTML tags, and I
only want the text. Any solution for this? Thanks !


Answer: The easiest option would be to
[`extract`](http://doc.scrapy.org/en/latest/topics/selectors.html#scrapy.selector.Selector.extract)
`//body//text()` and
[`join`](https://docs.python.org/2/library/string.html#string.join) everything
found:

    
    
    ''.join(sel.select("//body//text()").extract()).strip()
    

where `sel` is a
[`Selector`](http://doc.scrapy.org/en/latest/topics/selectors.html#scrapy.selector.Selector)
instance.

Another option is to use [`nltk`](https://pypi.python.org/pypi/nltk)'s
`clean_html()`:

    
    
    >>> import nltk
    >>> html = """
    ... <div class="post-text" itemprop="description">
    ... 
    ...         <p>I would like to have all the text visible from a website, after the HTML is rendered. I'm working in Python with Scrapy framework.
    ... With <code>xpath('//body//text()')</code> I'm able to get it, but with the HTML tags, and I only want the text. Any solution for this? Thanks !</p>
    ... 
    ...     </div>"""
    >>> nltk.clean_html(html)
    "I would like to have all the text visible from a website, after the HTML is rendered. I'm working in Python with Scrapy framework.\nWith xpath('//body//text()') I'm able to get it, but with the HTML tags, and I only want the text. Any solution for this? Thanks !"
    

Another option is to use
[`BeautifulSoup`](http://www.crummy.com/software/BeautifulSoup/bs4/doc/#get-
text)'s `get_text()`:

> `get_text()`
>
> If you only want the text part of a document or tag, you can use the
> `get_text()` method. It returns all the text in a document or beneath a tag,
> as a single Unicode string.
    
    
    >>> from bs4 import BeautifulSoup
    >>> soup = BeautifulSoup(html)
    >>> print soup.get_text().strip()
    I would like to have all the text visible from a website, after the HTML is rendered. I'm working in Python with Scrapy framework.
    With xpath('//body//text()') I'm able to get it, but with the HTML tags, and I only want the text. Any solution for this? Thanks !
    

Another option is to use [`lxml.html`](http://lxml.de/lxmlhtml.html)'s
`text_content()`:

> `.text_content()`
>
> Returns the text content of the element, including the text content of its
> children, with no markup.
    
    
    >>> import lxml.html
    >>> tree = lxml.html.fromstring(html)
    >>> print tree.text_content().strip()
    I would like to have all the text visible from a website, after the HTML is rendered. I'm working in Python with Scrapy framework.
    With xpath('//body//text()') I'm able to get it, but with the HTML tags, and I only want the text. Any solution for this? Thanks !
    

