<p dir="auto">While processing a large set of requests, <code class="notranslate">apparent_encoding()</code> works well for quite a while but eventually hangs. All I have at this point is a traceback:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="File &quot;/opt/wsgi/.../ENV/lib/python2.7/site-packages/requests/models.py&quot;, line 637, in apparent_encoding
return chardet.detect(self.content)['encoding']
File &quot;/opt/wsgi/.../ENV/lib/python2.7/site-packages/requests/packages/chardet/__init__.py&quot;, line 30, in detect
u.feed(aBuf)
File &quot;/opt/wsgi/.../ENV/lib/python2.7/site-packages/requests/packages/chardet/universaldetector.py&quot;, line 128, in feed
if prober.feed(aBuf) == constants.eFoundIt:
File &quot;/opt/wsgi/.../ENV/lib/python2.7/site-packages/requests/packages/chardet/charsetgroupprober.py&quot;, line 64, in feed
st = prober.feed(aBuf)
File &quot;/opt/wsgi/.../ENV/lib/python2.7/site-packages/requests/packages/chardet/sjisprober.py&quot;, line 74, in feed
self._mContextAnalyzer.feed(aBuf[i + 1 - charLen:i + 3"><pre class="notranslate"><code class="notranslate">File "/opt/wsgi/.../ENV/lib/python2.7/site-packages/requests/models.py", line 637, in apparent_encoding
return chardet.detect(self.content)['encoding']
File "/opt/wsgi/.../ENV/lib/python2.7/site-packages/requests/packages/chardet/__init__.py", line 30, in detect
u.feed(aBuf)
File "/opt/wsgi/.../ENV/lib/python2.7/site-packages/requests/packages/chardet/universaldetector.py", line 128, in feed
if prober.feed(aBuf) == constants.eFoundIt:
File "/opt/wsgi/.../ENV/lib/python2.7/site-packages/requests/packages/chardet/charsetgroupprober.py", line 64, in feed
st = prober.feed(aBuf)
File "/opt/wsgi/.../ENV/lib/python2.7/site-packages/requests/packages/chardet/sjisprober.py", line 74, in feed
self._mContextAnalyzer.feed(aBuf[i + 1 - charLen:i + 3
</code></pre></div>
<p dir="auto">I appreciate anything you can do to help.</p>
<p dir="auto">I am using chardet as part of a web crawler written in python3. I noticed that over time (many hours), the program consumes all memory. I narrowed down the problem to a single call of chardet.detect() method for certain web pages.</p>
<p dir="auto">After some testing, it seems that chardet has problem with some special input and I managed to get a sample of such an input. It consumes on my machine about 220 MB of memory (however, the input is 2.5 MB) and takes about 1:22 minutes to process (in contrast to 43 ms when the file is truncated to about 2 MB). It seems not to be limited to python3, in python2 the memory consumption is even worse (312 MB).</p>
<h5 dir="auto">Versions:</h5>
<p dir="auto">Fedora release 20 (Heisenbug) x86_64<br>
chardet-2.2.1 (via pip)<br>
python3-3.3.2-11.fc20.x86_64<br>
python-2.7.5-11.fc20.x86_64</p>
<h5 dir="auto">How to reproduce:</h5>
<p dir="auto">I cannot attach any files to this issue so I uploaded them to my dropbox account: <a href="https://www.dropbox.com/sh/26dry8zj18cv0m1/sKgP_E44qx/chardet_test.zip" rel="nofollow">https://www.dropbox.com/sh/26dry8zj18cv0m1/sKgP_E44qx/chardet_test.zip</a> Please let me know of a better place where to put it if necessary. Here is an overview of the content and the results:</p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="setup='import chardet; html = open(&quot;mem_leak_html.txt&quot;, &quot;rb&quot;).read()'
python3 -m timeit -s &quot;$setup&quot;  'chardet.detect(html[:2543482])'
# produces: 10 loops, best of 3: 43 ms per loop
python3 -m timeit -s &quot;$setup&quot;  'chardet.detect(html[:2543483])'
# produces: 1 loops, best of 3: 1min 22s per loop
python3 mem_leak_test.py
# produces:
# Good input left 2.65 MB of unfreed memory.
# Bad input left 220.16 MB of unfreed memory.

python -m timeit -s &quot;$setup&quot;  'chardet.detect(html[:2543482])'
# produces: 10 loops, best of 3: 41.7 ms per loop
python -m timeit -s &quot;$setup&quot;  'chardet.detect(html[:2543483])'
# produces: 10 loops, best of 3: 111 sec per loop
python mem_leak_test.py
# produces:
# Good input left 3.00 MB of unfreed memory.
# Bad input left 312.00 MB of unfreed memory."><pre class="notranslate">setup=<span class="pl-s"><span class="pl-pds">'</span>import chardet; html = open("mem_leak_html.txt", "rb").read()<span class="pl-pds">'</span></span>
python3 -m timeit -s <span class="pl-s"><span class="pl-pds">"</span><span class="pl-smi">$setup</span><span class="pl-pds">"</span></span>  <span class="pl-s"><span class="pl-pds">'</span>chardet.detect(html[:2543482])<span class="pl-pds">'</span></span>
<span class="pl-c"><span class="pl-c">#</span> produces: 10 loops, best of 3: 43 ms per loop</span>
python3 -m timeit -s <span class="pl-s"><span class="pl-pds">"</span><span class="pl-smi">$setup</span><span class="pl-pds">"</span></span>  <span class="pl-s"><span class="pl-pds">'</span>chardet.detect(html[:2543483])<span class="pl-pds">'</span></span>
<span class="pl-c"><span class="pl-c">#</span> produces: 1 loops, best of 3: 1min 22s per loop</span>
python3 mem_leak_test.py
<span class="pl-c"><span class="pl-c">#</span> produces:</span>
<span class="pl-c"><span class="pl-c">#</span> Good input left 2.65 MB of unfreed memory.</span>
<span class="pl-c"><span class="pl-c">#</span> Bad input left 220.16 MB of unfreed memory.</span>

python -m timeit -s <span class="pl-s"><span class="pl-pds">"</span><span class="pl-smi">$setup</span><span class="pl-pds">"</span></span>  <span class="pl-s"><span class="pl-pds">'</span>chardet.detect(html[:2543482])<span class="pl-pds">'</span></span>
<span class="pl-c"><span class="pl-c">#</span> produces: 10 loops, best of 3: 41.7 ms per loop</span>
python -m timeit -s <span class="pl-s"><span class="pl-pds">"</span><span class="pl-smi">$setup</span><span class="pl-pds">"</span></span>  <span class="pl-s"><span class="pl-pds">'</span>chardet.detect(html[:2543483])<span class="pl-pds">'</span></span>
<span class="pl-c"><span class="pl-c">#</span> produces: 10 loops, best of 3: 111 sec per loop</span>
python mem_leak_test.py
<span class="pl-c"><span class="pl-c">#</span> produces:</span>
<span class="pl-c"><span class="pl-c">#</span> Good input left 3.00 MB of unfreed memory.</span>
<span class="pl-c"><span class="pl-c">#</span> Bad input left 312.00 MB of unfreed memory.</span></pre></div>
<h6 dir="auto">mem_leak_test.py:</h6>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import resource
import chardet
import gc

mem_use = lambda: resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024
html = open(&quot;mem_leak_html.txt&quot;, &quot;rb&quot;).read()

def test(desc, instr):
    gc.collect()
    mem_start = mem_use()
    chardet.detect(instr)    
    gc.collect()
    mem_used = mem_use() - mem_start
    print('%s left %.2f MB of unfreed memory.' % (desc, mem_used))    

test('Good input', html[:2543482])
test('Bad input', html[:2543483])"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">resource</span>
<span class="pl-k">import</span> <span class="pl-s1">chardet</span>
<span class="pl-k">import</span> <span class="pl-s1">gc</span>

<span class="pl-s1">mem_use</span> <span class="pl-c1">=</span> <span class="pl-k">lambda</span>: <span class="pl-s1">resource</span>.<span class="pl-en">getrusage</span>(<span class="pl-s1">resource</span>.<span class="pl-v">RUSAGE_SELF</span>).<span class="pl-s1">ru_maxrss</span> <span class="pl-c1">/</span> <span class="pl-c1">1024</span>
<span class="pl-s1">html</span> <span class="pl-c1">=</span> <span class="pl-en">open</span>(<span class="pl-s">"mem_leak_html.txt"</span>, <span class="pl-s">"rb"</span>).<span class="pl-en">read</span>()

<span class="pl-k">def</span> <span class="pl-en">test</span>(<span class="pl-s1">desc</span>, <span class="pl-s1">instr</span>):
    <span class="pl-s1">gc</span>.<span class="pl-en">collect</span>()
    <span class="pl-s1">mem_start</span> <span class="pl-c1">=</span> <span class="pl-en">mem_use</span>()
    <span class="pl-s1">chardet</span>.<span class="pl-en">detect</span>(<span class="pl-s1">instr</span>)    
    <span class="pl-s1">gc</span>.<span class="pl-en">collect</span>()
    <span class="pl-s1">mem_used</span> <span class="pl-c1">=</span> <span class="pl-en">mem_use</span>() <span class="pl-c1">-</span> <span class="pl-s1">mem_start</span>
    <span class="pl-en">print</span>(<span class="pl-s">'%s left %.2f MB of unfreed memory.'</span> <span class="pl-c1">%</span> (<span class="pl-s1">desc</span>, <span class="pl-s1">mem_used</span>))    

<span class="pl-en">test</span>(<span class="pl-s">'Good input'</span>, <span class="pl-s1">html</span>[:<span class="pl-c1">2543482</span>])
<span class="pl-en">test</span>(<span class="pl-s">'Bad input'</span>, <span class="pl-s1">html</span>[:<span class="pl-c1">2543483</span>])</pre></div>