title: Removing html tags when crawling wikipedia with python's urllib2 and Beautifulsoup
body: <p>I am trying to crawl wikipedia to get some data for text mining. I am using python's urllib2 and Beautifulsoup. My question is that: is there an easy way of getting rid of the unnecessary tags(like links 'a's or 'span's) from the text I read.</p>
  
  <p>for this scenario:</p>
  
  <pre><code>import urllib2
  from BeautifulSoup import *
  opener = urllib2.build_opener()
  opener.addheaders = [('User-agent', 'Mozilla/5.0')]
  infile = opener.open("http://en.wikipedia.org/w/index.php?title=data_mining&amp;printable=yes")pool = BeautifulSoup(infile.read())
  res=pool.findAll('div',attrs={'class' : 'mw-content-ltr'}) # to get to content directly
  paragrapgs=res[0].findAll("p") #get all paragraphs
  </code></pre>
  
  <p>I get the paragraphs with lots of reference tags like:</p>
  
  <p>paragrapgs[0] =</p>
  
  <pre><code>&lt;p&gt;&lt;b&gt;Data mining&lt;/b&gt; (the analysis step of the &lt;b&gt;knowledge discovery in databases&lt;/b&gt; process,&lt;sup id="cite_ref-Fayyad_0-0" class="reference"&gt;&lt;a href="#cite_note-Fayyad-0"&gt;&lt;span&gt;[&lt;/span&gt;1&lt;span&gt;]&lt;/span&gt;&lt;/a&gt;&lt;/sup&gt; or KDD), a relatively young and interdisciplinary field of &lt;a href="/wiki/Computer_science" title="Computer science"&gt;computer science&lt;/a&gt;&lt;sup id="cite_ref-acm_1-0" class="reference"&gt;&lt;a href="#cite_note-acm-1"&gt;&lt;span&gt;[&lt;/span&gt;2&lt;span&gt;]&lt;/span&gt;&lt;/a&gt;&lt;/sup&gt;&lt;sup id="cite_ref-brittanica_2-0" class="reference"&gt;&lt;a href="#cite_note-brittanica-2"&gt;&lt;span&gt;[&lt;/span&gt;3&lt;span&gt;]&lt;/span&gt;&lt;/a&gt;&lt;/sup&gt; is the process of discovering new patterns from large &lt;a href="/wiki/Data_set" title="Data set"&gt;data sets&lt;/a&gt; involving methods at the intersection of &lt;a href="/wiki/Artificial_intelligence" title="Artificial intelligence"&gt;artificial intelligence&lt;/a&gt;, &lt;a href="/wiki/Machine_learning" title="Machine learning"&gt;machine learning&lt;/a&gt;, &lt;a href="/wiki/Statistics" title="Statistics"&gt;statistics&lt;/a&gt; and &lt;a href="/wiki/Database_system" title="Database system"&gt;database systems&lt;/a&gt;.&lt;sup id="cite_ref-acm_1-1" class="reference"&gt;&lt;a href="#cite_note-acm-1"&gt;&lt;span&gt;[&lt;/span&gt;2&lt;span&gt;]&lt;/span&gt;&lt;/a&gt;&lt;/sup&gt; The goal of data mining is to extract knowledge from a data set in a human-understandable structure&lt;sup id="cite_ref-acm_1-2" class="reference"&gt;&lt;a href="#cite_note-acm-1"&gt;&lt;span&gt;[&lt;/span&gt;2&lt;span&gt;]&lt;/span&gt;&lt;/a&gt;&lt;/sup&gt; and involves database and &lt;a href="/wiki/Data_management" title="Data management"&gt;data management&lt;/a&gt;, &lt;a href="/wiki/Data_Pre-processing" title="Data Pre-processing"&gt;data preprocessing&lt;/a&gt;, &lt;a href="/wiki/Statistical_model" title="Statistical model"&gt;model&lt;/a&gt; and &lt;a href="/wiki/Statistical_inference" title="Statistical inference"&gt;inference&lt;/a&gt; considerations, interestingness metrics, &lt;a href="/wiki/Computational_complexity_theory" title="Computational complexity theory"&gt;complexity&lt;/a&gt; considerations, post-processing of found structure, &lt;a href="/wiki/Data_visualization" title="Data visualization"&gt;visualization&lt;/a&gt; and &lt;a href="/wiki/Online_algorithm" title="Online algorithm"&gt;online updating&lt;/a&gt;.&lt;sup id="cite_ref-acm_1-3" class="reference"&gt;&lt;a href="#cite_note-acm-1"&gt;&lt;span&gt;[&lt;/span&gt;2&lt;span&gt;]&lt;/span&gt;&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;
  </code></pre>
  
  <p>Any ideas how to remove them and have pure text?</p>