Why is this lxml.etree.HTMLPullParser leaking memory?
Question: I'm trying to use lxml's HTMLPullParser on Linux Mint but I'm finding that the
memory usage keeps increasing and I'm not sure why. Here's my test code:

    
    
    # -*- coding: utf-8 -*-
    from __future__ import division, absolute_import, print_function, unicode_literals
    import lxml.etree
    import resource
    from io import DEFAULT_BUFFER_SIZE
    
    for _ in xrange(1000):
    with open('StackOverflow.html', 'r') as f:
        parser = lxml.etree.HTMLPullParser()
        while True:
            buf = f.read(DEFAULT_BUFFER_SIZE)
            if not buf: break
            parser.feed(buf)
        parser.close()
    
        # Print memory usage
        print((resource.getrusage(resource.RUSAGE_SELF)[2] * resource.getpagesize())/1000000.0)
    

StackOverflow.html is the homepage of stackoverflow that I've saved in the
same folder as the python script. I've tried adding explicit deletes and
clears but so far nothing has worked. What am I doing wrong?


Answer: Elements constructed by the parsers are leaking, and I can't see an API
contract violation in your code that's causing it. Since the objects survive a
manual garbage collection run with `gc.collect()`, your best bet is probably
to try a different parsing strategy as a workaround.

To see the root cause, I used the memory exploration module
[objgraph](http://mg.pov.lt/objgraph/) and installed
[xdot](http://pypi.python.org/pypi/xdot) to view the graphs it created.

Before running the code, I ran:

    
    
    In [3]: import objgraph
    
    In [4]: objgraph.show_growth()
    

After running the code, I ran:

    
    
    In [6]: objgraph.show_growth()
    tuple                  1616      +147
    _Element                146      +146
    list                   1100       +24
    wrapper_descriptor     1423       +15
    weakref                1155        +6
    getset_descriptor       677        +4
    dict                   2777        +4
    member_descriptor       315        +3
    method_descriptor       891        +2
    _TempStore                2        +1
    
    In [7]: import random
    
    In [8]: objgraph.show_chain(
       ...: objgraph.find_backref_chain(
       ...: random.choice(objgraph.by_type('_Element')), objgraph.is_proper_module))
    Graph written to /tmp/objgraph-bfuwa9.dot (8 nodes)
    Spawning graph viewer (xdot)
    

Note: the numbers might be different than what you see depending on the
webpage viewed.

