tornado AsyncHTTPClient.fetch exception
Question: I am using `tornado.httpclient.AsyncHTTPClient.fetch` to fetch domains from
list. When I put domains to fetch with some big interval(500 for example) all
works good, but when I decrease the inerval to 100, next exception occurs time
to time:

    
    
    Traceback (most recent call last):
      File "/home/crchemist/python-2.7.2/lib/python2.7/site-packages/tornado/simple_httpclient.py", line 289, in cleanup
        yield
      File "/home/crchemist/python-2.7.2/lib/python2.7/site-packages/tornado/stack_context.py", line 183, in wrapped
        callback(*args, **kwargs)
      File "/home/crchemist/python-2.7.2/lib/python2.7/site-packages/tornado/simple_httpclient.py", line 384, in _on_chunk_length
        self._on_chunk_data)
      File "/home/crchemist/python-2.7.2/lib/python2.7/site-packages/tornado/iostream.py", line 180, in read_bytes
        self._check_closed()
      File "/home/crchemist/python-2.7.2/lib/python2.7/site-packages/tornado/iostream.py", line 504, in _check_closed
        raise IOError("Stream is closed")
    IOError: Stream is closed
    

What can be the reason of this behavior? Code looks like this:

    
    
    def fetch_domain(domain):
        http_client = AsyncHTTPClient()
        request = HTTPRequest('http://' + domain,
           user_agent=CRAWLER_USER_AGENT)
        http_client.fetch(request, handle_domain)
    
    
    class DomainFetcher(object):
        def __init__(self, domains_iterator):
            self.domains = domains_iterator
    
        def __call__(self):
            try:
                domain = next(self.domains)
            except StopIteration:
                domain_generator.stop()
                ioloop.IOLoop.instance().stop()
            else:
                fetch_domain(domain)
    
    domain_generator = ioloop.PeriodicCallback(DomainFetcher(domains), 500)
    domain_generator.start()
    
    


Answer: note that `tornado.ioloop.PeriodicCallback` [takes a cycle time **in integer
ms**](http://www.tornadoweb.org/documentation/_modules/tornado/ioloop.html#PeriodicCallback)
while the
[`HTTPRequest`](http://www.tornadoweb.org/documentation/_modules/tornado/httpclient.html#HTTPRequest)
object is configured with a `connect_timeout` and/or a `request_timeout` of
float **seconds** ([see
doc](http://www.tornadoweb.org/documentation/httpclient.html)).

"_Users browsing the Internet feel that responses are "instant" when delays
are less than 100 ms from click to response_ " ([from
wikipedia](http://en.wikipedia.org/wiki/Network_performance#Relationship_between_latency_and_throughput))
See [this ServerFault question for **normal latency**
values](http://serverfault.com/questions/137348/how-much-network-latency-is-
typical-for-east-west-coast-usa).

`IOError: Stream is closed` is validly being raised to inform you that your
connection timed out without completing, or more accurately, you called the
callback manually on a pipe that wasn't open yet. This is good, since it is
not abnormal for latency to be > 100ms; if you expect your fetches to complete
reliably you should raise this value.

Once you've got your timeout set to something sane, consider wrapping your
fetches in a try/except retry loop as this is a normal exception _that you can
expect to occur in production_. Just be careful to set a retry limit!

* * *

Since you're using an async framework, why not let it handle the async
callback itself instead of running said callback on a fixed interval?
[Epoll/kqueue are efficient and supported by this
framework.](http://www.tornadoweb.org/documentation/overview.html?highlight=epoll%20kqueue)

    
    
    import ioloop
    
    def handle_request(response):
        if response.error:
            print "Error:", response.error
        else:
            print response.body
        ioloop.IOLoop.instance().stop()
    
    http_client = httpclient.AsyncHTTPClient()
    http_client.fetch("http://www.google.com/", handle_request)
    ioloop.IOLoop.instance().start()
    

^ Copied verbatim [from the
doc](http://www.tornadoweb.org/documentation/httpclient.html).

If you go this route, the only gotcha is to code your request queue so that
you have a maximum open connections enforced. Otherwise you're likely to end
up with a race condition when doing serious scraping.

It's been ~1yr since I touched Tornado myself, so please let me know if there
are inaccuracies in this response and I will revise.

