Predictive Analytics on distributed systems vs standalone system I have somewhat philosophical question regarding performing predictive analytics on distributed systems (such as hadoop). I am no expert on this subject so maybe other folks who have more advance knowledge on this subject can enlighten me. Here is where I am getting confused:

How can a statistical model (such as linear regression model) built on Hadoop where data is distributed across many nodes, and where work is done only on subset of original data-set residing in each node have same or better accuracy than linear regression model built on single server where all data resides locally. Would the error-rate (accuracy) of a model build on hadoop be same, better or worse than model build on local system. I was thinking that since all data reside in local system, algorithm can be tuned to optimize error in a linear regression model. However, in case of Hadoop where data is distributed, what if only local optimization lead to drastically bad algorithm overall? Are the algorithms written separately for distributed system to take into account data distribution?
There is nothing special about building something across many machines, or Hadoop, so maybe we can take that out of the question.

One general reason is that, even if the model building process were worse, building on more data could result in a better result.

You're asking why building many models on subsets of data could be better, but this is precisely why ensemble models like random forests can outperform single instances.

But lastly, it's not true that distributed algorithms always work by making an ensemble of small models, so the premise is not strictly true.