title: Is it possible for Scrapy to get plain text from raw HTML data?
body: <p>For example:</p>
  
  <pre><code>scrapy shell http://scrapy.org/
  content = hxs.select('//*[@id="content"]').extract()[0]
  print content
  </code></pre>
  
  <p>Then, I get the following raw HTML code:</p>
  
  <pre class="lang-html prettyprint-override"><code>&lt;div id="content"&gt;
  
  
    &lt;h2&gt;Welcome to Scrapy&lt;/h2&gt;
  
    &lt;h3&gt;What is Scrapy?&lt;/h3&gt;
  
    &lt;p&gt;Scrapy is a fast high-level screen scraping and web crawling
      framework, used to crawl websites and extract structured data from their
      pages. It can be used for a wide range of purposes, from data mining to
      monitoring and automated testing.&lt;/p&gt;
  
    &lt;h3&gt;Features&lt;/h3&gt;
  
    &lt;dl&gt;
  
      &lt;dt&gt;Simple&lt;/dt&gt;
      &lt;dt&gt;
      &lt;/dt&gt;
      &lt;dd&gt;Scrapy was designed with simplicity in mind, by providing the features
        you need without getting in your way
      &lt;/dd&gt;
  
      &lt;dt&gt;Productive&lt;/dt&gt;
      &lt;dd&gt;Just write the rules to extract the data from web pages and let Scrapy
        crawl the entire web site for you
      &lt;/dd&gt;
  
      &lt;dt&gt;Fast&lt;/dt&gt;
      &lt;dd&gt;Scrapy is used in production crawlers to completely scrape more than
        500 retailer sites daily, all in one server
      &lt;/dd&gt;
  
      &lt;dt&gt;Extensible&lt;/dt&gt;
      &lt;dd&gt;Scrapy was designed with extensibility in mind and so it provides
        several mechanisms to plug new code without having to touch the framework
        core
  
      &lt;/dd&gt;
      &lt;dt&gt;Portable, open-source, 100% Python&lt;/dt&gt;
      &lt;dd&gt;Scrapy is completely written in Python and runs on Linux, Windows, Mac and BSD&lt;/dd&gt;
  
      &lt;dt&gt;Batteries included&lt;/dt&gt;
      &lt;dd&gt;Scrapy comes with lots of functionality built in. Check &lt;a
          href="http://doc.scrapy.org/en/latest/intro/overview.html#what-else"&gt;this
        section&lt;/a&gt; of the documentation for a list of them.
      &lt;/dd&gt;
  
      &lt;dt&gt;Well-documented &amp;amp; well-tested&lt;/dt&gt;
      &lt;dd&gt;Scrapy is &lt;a href="/doc/"&gt;extensively documented&lt;/a&gt; and has an comprehensive test suite
        with &lt;a href="http://static.scrapy.org/coverage-report/"&gt;very good code
          coverage&lt;/a&gt;&lt;/dd&gt;
  
      &lt;dt&gt;&lt;a href="/community"&gt;Healthy community&lt;/a&gt;&lt;/dt&gt;
      &lt;dd&gt;
        1,500 watchers, 350 forks on Github (&lt;a href="https://github.com/scrapy/scrapy"&gt;link&lt;/a&gt;)&lt;br&gt;
        700 followers on Twitter (&lt;a href="http://twitter.com/ScrapyProject"&gt;link&lt;/a&gt;)&lt;br&gt;
        850 questions on StackOverflow (&lt;a href="http://stackoverflow.com/tags/scrapy/info"&gt;link&lt;/a&gt;)&lt;br&gt;
        200 messages per month on mailing list (&lt;a
          href="https://groups.google.com/forum/?fromgroups#!aboutgroup/scrapy-users"&gt;link&lt;/a&gt;)&lt;br&gt;
        40-50 users always connected to IRC channel (&lt;a href="http://webchat.freenode.net/?channels=scrapy"&gt;link&lt;/a&gt;)
      &lt;/dd&gt;
  
      &lt;dt&gt;&lt;a href="/support"&gt;Commercial support&lt;/a&gt;&lt;/dt&gt;
      &lt;dd&gt;A few companies provide Scrapy consulting and support&lt;/dd&gt;
  
      &lt;p&gt;Still not sure if Scrapy is what you're looking for?. Check out &lt;a
          href="http://doc.scrapy.org/en/latest/intro/overview.html"&gt;Scrapy at a
        glance&lt;/a&gt;.
  
      &lt;/p&gt;
      &lt;h3&gt;Companies using Scrapy&lt;/h3&gt;
  
      &lt;p&gt;Scrapy is being used in large production environments, to crawl
        thousands of sites daily. Here is a list of &lt;a href="/companies/"&gt;Companies
          using Scrapy&lt;/a&gt;.&lt;/p&gt;
  
      &lt;h3&gt;Where to start?&lt;/h3&gt;
  
      &lt;p&gt;Start by reading &lt;a href="http://doc.scrapy.org/en/latest/intro/overview.html"&gt;Scrapy at a glance&lt;/a&gt;,
        then &lt;a href="/download/"&gt;download Scrapy&lt;/a&gt; and follow the &lt;a
            href="http://doc.scrapy.org/en/latest/intro/tutorial.html"&gt;Tutorial&lt;/a&gt;.
  
  
      &lt;/p&gt;&lt;/dl&gt;
  &lt;/div&gt;
  </code></pre>
  
  <p>But I want to get <strong>plain text</strong> directly from scrapy.</p>
  
  <p>I do not want to use any xPath selectors to extract the <code>p</code>, <code>h2</code>, <code>h3</code>... tags, since I am crawling a website whose main content is embedded into a <code>table</code>, <code>tbody</code>; recursively. It can be a tedious task to find the xPath.</p>
  
  <p>Can this be implemented by a built in function in Scrapy? Or do I need external tools to convert it? I have read through all of Scrapy's docs, but have gained nothing.</p>
  
  <p>This is a sample site which can convert raw HTML into plain text: <a href="http://beaker.mailchimp.com/html-to-text" rel="nofollow noreferrer">http://beaker.mailchimp.com/html-to-text</a></p>