Process Memory grows huge -Tornado CurlAsyncHTTPClient
Question: I am using Tornado CurlAsyncHTTPClient. My process memory keeps growing for
both blocking and non blocking requests when I instantiate corresponding
httpclients for each request. This memory usage growth does not happen if I
just have one instance of the
httpclients(tornado.httpclient.HTTPClient/tornado.httpclient.AsyncHTTPClient)
and reuse them.

Also If I use SimpleAsyncHTTPClient instead of CurlAsyncHTTPClient this memory
growth doesnot happen irrespective of how I instantiate.

Here is a sample code that reproduces this,

    
    
    import tornado.httpclient
    import json
    import functools
    
    instantiate_once = False
    tornado.httpclient.AsyncHTTPClient.configure('tornado.curl_httpclient.CurlAsyncHTTPClient')
    
    hc, io_loop, async_hc = None, None, None
    if instantiate_once:
        hc = tornado.httpclient.HTTPClient()
        io_loop = tornado.ioloop.IOLoop()
        async_hc = tornado.httpclient.AsyncHTTPClient(io_loop=io_loop)
    
    def fire_sync_request():
        global count
        if instantiate_once:
            global hc
        if not instantiate_once:
            hc = tornado.httpclient.HTTPClient()
        url = '<Please try with a url>'
        try:
            resp = hc.fetch(url)
        except (Exception,tornado.httpclient.HTTPError) as e:
            print str(e)
        if not instantiate_once:
            hc.close()
    
    def fire_async_requests():
        #generic response callback fn
        def response_callback(response):
            response_callback_info['response_count'] += 1
            if response_callback_info['response_count'] >= request_count:
                io_loop.stop()
        if instantiate_once:
            global io_loop, async_hc
        if not instantiate_once:
            io_loop = tornado.ioloop.IOLoop()
        requests = ['<Please add ur url to try>']*5
        response_callback_info = {'response_count': 0}
        request_count = len(requests)
        global count
        count +=request_count
        hcs=[]
        for url in requests:
            kwargs ={}
            kwargs['method'] = 'GET'
           if not instantiate_once:
                async_hc = tornado.httpclient.AsyncHTTPClient(io_loop=io_loop)
            async_hc.fetch(url, callback=functools.partial(response_callback), **kwargs)
            if not instantiate_once:
            hcs.append(async_hc)
    
        io_loop.start()
        for hc in hcs:
            hc.close()
        if not instantiate_once:
            io_loop.close()
    
    if __name__ == '__main__':
        import sys
        if sys.argv[1] == 'sync':
            while True:
                output = fire_sync_request()
        elif sys.argv[1] == 'async':
            while True:
                output = fire_async_requests()
    

Here set instantiate_once variable to True, and execute python check.py sync
or python check.py async. The process memory increases continuously

With instantiate_once=False, this doesnot happen.

Also If I use SimpleAsyncHTTPClient instead of CurlAsyncHTTPClient this memory
growth doesnot happen.

I have python 2.7/ tornado 2.3.2/ pycurl(libcurl/7.26.0 GnuTLS/2.12.20
zlib/1.2.7 libidn/1.25 libssh2/1.4.2 librtmp/2.3)

I could reproduce the same issue with latest tornado 3.2

Please help me to understand this behaviour and figure out the right way of
using tornado as http library.


Answer: HTTPClient and AsyncHTTPClient are designed to be reused, so it will always be
more efficient not to recreate them all the time. In fact, AsyncHTTPClient
will try to magically detect if there is an existing AsyncHTTPClient on the
same IOLoop and use that instead of creating a new one.

But even though it's better to reuse one http client object, it shouldn't leak
to create many of them as you're doing here (as long as you're closing them).
This looks like a bug in pycurl: <https://github.com/pycurl/pycurl/issues/182>

