title: Is scikit-learn suitable for big data tasks?
body: <p>I'm working on a TREC task involving use of machine learning techniques, where the dataset consists of more than 5 terabytes of web documents, from which bag-of-words vectors are planned to be extracted. <code>scikit-learn</code> has a nice set of functionalities that seems to fit my need, but I don't know whether it is going to scale well to handle big data. For example, is <code>HashingVectorizer</code> able to handle 5 terabytes of documents, and is it feasible to parallelize it? Moreover, what are some alternatives out there for large-scale machine learning tasks?</p>