<p dir="auto">On one of the pages that I'm parsing I got:</p>
<p dir="auto">File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 777, in text<br>
content = str(self.content, encoding, errors='replace')<br>
LookupError: unknown encoding: ISO-LATIN-1</p>
<p dir="auto">I think it would be more sensible to try a default encoding (utf-8 and windows-1252 as a fallback) when Python doesn't support the encoding defined in the page.</p>
<p dir="auto">Also, using encoding.lower().replace('-latin-', '-8859-') will go a long way fixing the common case of pages using iso-latin-N instead of iso-8859-N.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; url = 'http://dilbert.com'
&gt;&gt;&gt; len(urllib2.urlopen(url).read())
30194
&gt;&gt;&gt; len(requests.get(url).content)
Traceback (most recent call last):
  File &quot;&lt;stdin&gt;&quot;, line 1, in &lt;module&gt;
  File &quot;build/bdist.macosx-10.7-intel/egg/requests/models.py&quot;, line 713, in content
LookupError: unknown encoding: utf-8lias
&gt;&gt;&gt; "><pre class="notranslate"><code class="notranslate">&gt;&gt;&gt; url = 'http://dilbert.com'
&gt;&gt;&gt; len(urllib2.urlopen(url).read())
30194
&gt;&gt;&gt; len(requests.get(url).content)
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "build/bdist.macosx-10.7-intel/egg/requests/models.py", line 713, in content
LookupError: unknown encoding: utf-8lias
&gt;&gt;&gt; 
</code></pre></div>