Using sklearn DictVectorizer in real-time systems
If you're doing anything at the millions of records per second level, you're going to need something more high-performance than Python. Spark/PySpark is ideal for this since it interfaces with your existing Python knowledge and supports HDFS/distributed technologies that are made to take on the millions per second sized tasks.

As for sklearn: you might be out of luck here. The fit_transform() or fit() methods necessarily have to consider every record that you pass to them to keep the hash up to date, so if you have millions of data points coming in you're going to have to do millions of computations to keep moving, whether you do them one at a time or in a batch. The source code for the transform() method should convince you that doing either is equivalent.

One caveat is that if you have predictable features, e.g. maybe you're receiving counts of foo's bar's and baz's, like in the first link:

D = [{'foo': 1, 'bar': 2}, {'foo': 3, 'baz': 1}]


Then you don't need to update the hash every time, all you have to do is transform D into the vectorized format and add it to your collection. As long as you don't need to fit new features this can be done relatively fast. However, if you're doing something less structured like NLP this won't give you hardly any useful information.