Realtime data processing in Django/Python
Question: We are working on a project involving realtime data processing. We plan to use
Django/Python. The actual process is:

  1. Tens of thousands of devices take 4 samples per seconds (0, 0.25, 0.5, 0.75) and continuously send back to our Django server, basically they are time series with timestamp and value
  2. We need to align samples from all devices according to the timestamp (need to have milliseconds precision) and do a simple average of all the time series
  3. All these needs to be done in realtime (maximum 1 second delay) and send away using another thread

We are looking into RRDTool and scikits.timeseries, but they don't have the
precision of milliseconds, so they couldn't align our time series.

Just wondering is there any tools/data structure we can use with Django/Python
for this type of realtime data processing. And thread safe is important, as
sending the result away will be done in another thread.

Thanks in advance.


Answer: Your options for real time web services in python are:
[Twisted](http://twistedmatrix.com/trac/),
[Tornado](http://www.tornadoweb.org/) and [Eventlet](http://eventlet.net/)

You can integrate all this to work with Python/Django. [Tutorial on
that](http://lincolnloop.com/blog/2009/sep/15/using-django-inside-tornado-web-
server/).

