title: Boilerpipe python keeps running into non html files
body: <p>so I'm using boilerpipe to extract html from urls I get as results from a search query. But I keep running into non html sites and I'm not sure how to work around it. At first I installed regex to look for .pdf in the url but preferably it would still read the pdf file somehow. Anyways does anyone know a workaround for this? Here is my code if you need it :</p>
  <pre><code>def searchgoogle(query, num):
      list = []
      for j in search(query, num=num, stop=num):
        list.append(j)
      print(len(list))
      return list
  
    b = searchgoogle(query, num)
    print(b)
    for i in b:
        print (i)
        extractor = extractors.ArticleExtractor()
        headers = {
            'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) '
                          'Chrome/50.0.2661.102 Safari/537.36'
        }
        url = i
        extractor = extractors.ArticleExtractor()
        if re.search(&quot;.pdf&quot;,i) or re.search(&quot;send_file&quot;,i):
          print(&quot;nope&quot;)
        else:
          resp = requests.get(url, headers=headers)
          if resp.ok:
              doc = extractor.get_content(resp.text)
              comparetexts(sustext,doc)
  </code></pre>
  <p>Thanks so much!</p>