title: Python Newspaper with web archive (wayback machine)
body: <p>I'm trying to use the Python library <a href="https://github.com/codelucas/newspaper" rel="nofollow noreferrer">newspaper</a> with the archives from the <a href="https://archive.org/web/" rel="nofollow noreferrer">Wayback Machine</a>, which stores old versions of websites that were archived. Theoretically, old news articles could be queried and downloaded from these archives.</p>
  <p>For instance, the follow code queries the archives for <strong>CNBC</strong> for a specific archive date.</p>
  <pre><code>import newspaper
  url = 'http://web.archive.org/web/20161201123529/http://www.cnbc.com/'
  paper = newspaper.build(url, memoize_articles = False )
  </code></pre>
  <p>Although the archived website itself contains links to actual news articles from 2016-12-01, the newspaper module does not seem to pick them up. Instead, you get urls such as:</p>
  <pre><code>https://blog.archive.org/2016/10/23/defining-web-pages-web-sites-and-web-captures/
  </code></pre>
  <p>which are not actual articles from this archived version of CNBC. However, newspaper works great with <em>today</em>'s version of <a href="http://cnbc.com" rel="nofollow noreferrer">CNBC</a>.</p>
  <p>I suppose that it gets confused because of the format of the url (which contains two <code>http</code>s).  Does anyone have any suggestions on how to extract articles from the <a href="https://archive.org/web/" rel="nofollow noreferrer">Wayback Machine</a> archives?</p>