title: How to scale a large scale data in scikit-learn?
body: <p>The whole data set has <code>80</code> million samples, each sample have <code>200</code> dense features. We often train a classifier with batch processing. For example, we adopt the <code>clf = sklearn.linear_model.SGDClassifier</code>, then we can use <code>clf.partial_fit(batch_data, batch_y)</code> to fit the model with the batch data. </p>
  
  <p>Before that, we should first scale the <code>batch_data</code>. Suppose we use the <code>mean-std</code> normalization. So we should obtain the global mean and standard deviations for each feature dimension. After that, we can use the global mean and stds to scale the batch_data. </p>
  
  <p>Now the problem is how to obtain the mean and std of the whole data set. To compute the global std, we could use $\sigma^2 = E(X^2) - E(X)^2$. Then we should compute the <code>E(X^2)</code> and <code>E(X)</code> by batch processing. </p>
  
  <p>I think the <code>Hadoop</code> or <code>Spark</code> might be suitable for this task. For each batch of data, we could start a instance to compute the partial <code>E(X^2)</code> and <code>E(X)</code>, then reduce them into the global ones.</p>
  
  <p>In <code>scikit-learn</code>, is there any more efficient way to scale the large data set? Maybe we could use the <code>multithreading</code> or start multi processes to handle batch data, then reduce the results to get the global means and stds.</p>