IOError: request data read error
Question: I seem to be getting an IOError: request data read error quite a lot when i'm
doing an Ajax upload. For example out of every 5 file uploads it errors out on
atleast 3.

Other people seem to have had the same issue. Eg.

  * <http://stackoverflow.com/questions/2641665/django-upload-failing-on-request-data-read-error>
  * <http://stackoverflow.com/questions/411902/django-file-upload-failing-occasionally>

Some other observations:

  * It's definitely not my internet connection or a browser issue. Seems to be happening on all browsers chrome/FF/opera.

  * I'm running **django 1.1.1 Apache/2.2.14 (Ubuntu) mod_ssl/2.2.14 OpenSSL/0.9.8k mod_wsgi/2.8 Python/2.6.5** on Lucid.

  * It is also not the file size. I can sometimes upload 1+ MB files but fail on 180 Kb files.

* * *

**Traceback**

    
    
    Traceback (most recent call last):
    
      File "/home/ubuntu/.virtualenvs/anonymous_app/lib/python2.6/site-packages/django/core/handlers/base.py", line 98, in get_response
        response = middleware_method(request, e)
    
      File "/home/ubuntu/.virtualenvs/anonymous_app/lib/python2.6/site-packages/django/core/handlers/base.py", line 92, in get_response
        response = callback(request, *callback_args, **callback_kwargs)
    
      File "/home/ubuntu/.virtualenvs/anonymous_app/lib/python2.6/site-packages/django/contrib/auth/decorators.py", line 78, in __call__
        return self.view_func(request, *args, **kwargs)
    
      File "/home/ubuntu/webapps/anonymous_app/app/do_work/views/__init__.py", line 391, in some_form_ajax_upload
        f = request.FILES.get('file_upload')
    
      File "/home/ubuntu/.virtualenvs/anonymous_app/lib/python2.6/site-packages/django/core/handlers/wsgi.py", line 187, in _get_files
        self._load_post_and_files()
    
      File "/home/ubuntu/.virtualenvs/anonymous_app/lib/python2.6/site-packages/django/core/handlers/wsgi.py", line 137, in _load_post_and_files
        self._post, self._files = self.parse_file_upload(self.META, self.environ['wsgi.input'])
    
      File "/home/ubuntu/.virtualenvs/anonymous_app/lib/python2.6/site-packages/django/http/__init__.py", line 124, in parse_file_upload
        return parser.parse()
    
      File "/home/ubuntu/.virtualenvs/anonymous_app/lib/python2.6/site-packages/django/http/multipartparser.py", line 133, in parse
        for item_type, meta_data, field_stream in Parser(stream, self._boundary):
    
      File "/home/ubuntu/.virtualenvs/anonymous_app/lib/python2.6/site-packages/django/http/multipartparser.py", line 606, in __iter__
        for sub_stream in boundarystream:
    
      File "/home/ubuntu/.virtualenvs/anonymous_app/lib/python2.6/site-packages/django/http/multipartparser.py", line 420, in next
        return LazyStream(BoundaryIter(self._stream, self._boundary))
    
      File "/home/ubuntu/.virtualenvs/anonymous_app/lib/python2.6/site-packages/django/http/multipartparser.py", line 446, in __init__
        unused_char = self._stream.read(1)
    
      File "/home/ubuntu/.virtualenvs/anonymous_app/lib/python2.6/site-packages/django/http/multipartparser.py", line 299, in read
        out = ''.join(parts())
    
      File "/home/ubuntu/.virtualenvs/anonymous_app/lib/python2.6/site-packages/django/http/multipartparser.py", line 292, in parts
        chunk = self.next()
    
      File "/home/ubuntu/.virtualenvs/anonymous_app/lib/python2.6/site-packages/django/http/multipartparser.py", line 314, in next
        output = self._producer.next()
    
      File "/home/ubuntu/.virtualenvs/anonymous_app/lib/python2.6/site-packages/django/http/multipartparser.py", line 375, in next
        data = self.flo.read(self.chunk_size)
    
      File "/home/ubuntu/.virtualenvs/anonymous_app/lib/python2.6/site-packages/django/http/multipartparser.py", line 405, in read
        return self._file.read(num_bytes)
    
    IOError: request data read error
    
    
    <WSGIRequest
    GET:<QueryDict: {}>,
    POST:<could not parse>,
    COOKIES:{'__utma': '168279989.1688771210.1285773436.1285773436.1285773436.1',
     '__utmb': '168279989.20.10.1285773436',
     '__utmc': '168279989',
     '__utmz': '168279989.1285773436.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)',
     'beta': 'True',
     'sessionid': 'b1ecf92f2bba13e1885d07803e10aa03',
     'timezone_offset': '-330'},
    META:{'CONTENT_LENGTH': '188575',
     'CONTENT_TYPE': 'multipart/form-data; boundary=---------------------------57602381214905740261171925981',
     'DOCUMENT_ROOT': '/htdocs',
     'GATEWAY_INTERFACE': 'CGI/1.1',
     'HTTPS': '1',
     'HTTP_ACCEPT': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
     'HTTP_ACCEPT_CHARSET': 'ISO-8859-1,utf-8;q=0.7,*;q=0.7',
     'HTTP_ACCEPT_ENCODING': 'gzip,deflate',
     'HTTP_ACCEPT_LANGUAGE': 'en-us,en;q=0.5',
     'HTTP_CONNECTION': 'keep-alive',
     'HTTP_COOKIE': 'beta=True; __utma=168279989.1688771210.1285773436.1285773436.1285773436.1; __utmb=168279989.20.10.1285773436; __utmc=168279989; __utmz=168279989.1285773436.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); sessionid=b1ecf92f2bba13e1885d07803e10aa03; timezone_offset=-330',
     'HTTP_HOST': 'xxxxxx.compute-1.amazonaws.com',
     'HTTP_KEEP_ALIVE': '115',
     'HTTP_REFERER': 'https://ec2-184-72-79-96.compute-1.amazonaws.com/do-my-somees/enter/some-documents/',
     'HTTP_USER_AGENT': 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.10) Gecko/20100915 Ubuntu/10.04 (lucid) Firefox/3.6.10',
     'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/X11R6/bin',
     'PATH_INFO': u'/do-my-somees/enter/some-documents/ajax-upload/Other-some-Document/',
     'PATH_TRANSLATED': '/home/ubuntu/webapps/anonymous_app/settings/apache/qa.wsgi.py/do-my-somees/enter/some-documents/ajax-upload/Other-some-Document/',
     'QUERY_STRING': '',
     'REMOTE_ADDR': '',
     'REMOTE_PORT': '15561',
     'REQUEST_METHOD': 'POST',
     'REQUEST_URI': '/do-my-somees/enter/some-documents/ajax-upload/Other-some-Document/',
     'SCRIPT_FILENAME': '/home/ubuntu/webapps/anonymous_app/settings/apache/qa.wsgi.py',
     'SCRIPT_NAME': u'',
     'SERVER_ADDR': '10.196.142.182',
     'SERVER_ADMIN': 'dev@anonymous_app.com',
     'SERVER_NAME': 'ec2-184-72-79-96.compute-1.amazonaws.com',
     'SERVER_PORT': '443',
     'SERVER_PROTOCOL': 'HTTP/1.1',
     'SERVER_SIGNATURE': '<address>Apache/2.2.14 (Ubuntu) Server at ec2-184-72-79-96.compute-1.amazonaws.com Port 443</address>\n',
     'SERVER_SOFTWARE': 'Apache/2.2.14 (Ubuntu)',
     'SSL_TLS_SNI': 'ec2-184-72-79-96.compute-1.amazonaws.com',
     'mod_wsgi.application_group': 'qa.anonymous_app.com|',
     'mod_wsgi.callable_object': 'application',
     'mod_wsgi.listener_host': '',
     'mod_wsgi.listener_port': '443',
     'mod_wsgi.process_group': '',
     'mod_wsgi.reload_mechanism': '0',
     'mod_wsgi.script_reloading': '1',
     'mod_wsgi.version': (2, 8),
     'wsgi.errors': <mod_wsgi.Log object at 0xb9456860>,
     'wsgi.file_wrapper': <built-in method file_wrapper of mod_wsgi.Adapter object at 0xb936a968>,
     'wsgi.input': <mod_wsgi.Input object at 0xb9720e30>,
     'wsgi.multiprocess': True,
     'wsgi.multithread': False,
     'wsgi.run_once': False,
     'wsgi.url_scheme': 'https',
     'wsgi.version': (1, 0)}>
    


Answer: I get this exception, too. In the Apache error logfile I see this:

    
    
    [Wed Aug 17 08:30:45 2011] [error] [client 10.114.48.206] (70014)End of file found: mod_wsgi (pid=9722): Unable to get bucket brigade for request., referer: https://egs-work/modwork/beleg/188074/edit/
    [Wed Aug 17 08:30:45 2011] [error] [client 10.114.48.206] mod_wsgi (pid=3572): Exception occurred processing WSGI script '/home/modwork_egs_p/modwork_egs/apache/django_wsgi.py'.
    [Wed Aug 17 08:30:45 2011] [error] [client 10.114.48.206] IOError: failed to write data
    

Versions:

    
    
    apache2-prefork-2.2.15-3.7.x86_64
    apache2-mod_wsgi-3.3-1.8.x86_64 WSGIDaemonProcess with threads=1
    mod_ssl/2.2.15
    Linux egs-work 2.6.34.8-0.2-default #1 SMP 2011-04-06 18:11:26 +0200 x86_64 x86_64 x86_64 GNU/Linux
    openSUSE 11.3 (x86_64)
    

First I was confused, because the last line "failed to **write** data" does
not fit to the django code "load post data". But I guess that django wants to
write an error page to the client. But the client has canceled the tcp
connection. And now http 500 page can't be written to the client.

The client disconnected after sending the request, and before getting the
response:

  * The user closed the browser or navigated to an other page.
  * The user pressed the reload button.

I have seen this only with POST-Requests (not GET). If POST is used, the
webserver does read at least twice: First to get the headers, the second to
get the data. The second read fails.

It is easy to reproduce:

Insert some code which waits before the first access to request.POST happens
(be sure, that no middleware accesses request.POST before time.sleep()):

    
    
    def edit(request):
        import time
        time.sleep(3)
        #.....
    

Now do a big POST (e.g. file upload). I don't know the apache buffer size. But
5 MB should be enough. When the browser shows the hourglass, browse to an
other page. The browser will cancel the request and the exception should be in
the logfile.

This is my Middleware, since I don't want to get the above traceback in our
logfiles:

    
    
    class HandleExceptionMiddleware:
    
        def process_exception(self, request, exception):
            if isinstance(exception, IOError) and 'request data read error' in unicode(exception):
                logging.info('%s %s: %s: Request was canceled by the client.' % (
                        request.build_absolute_uri(), request.user, exception))
                return HttpResponseServerError()
    

