The growth in the use of computationally intensive statistical procedures, especially with Big Data, has necessitated the usage of parallel computation on diverse platforms such as multicore, GPU, clusters and clouds. However, slowdown due to interprocess communication costs typically limits such methods to "embarrassingly parallel" (EP) algorithms, especially on non-shared memory platforms. This paper develops a broadly-applicable method for converting many non-EP algorithms into statistically equivalent EP ones. The method is shown to yield excellent levels of speedup for a variety of statistical computations. It also overcomes certain problems of memory limitations.
@cite_1: An implementation of the backfitting algorithm for generalised additive models which is suitable for parallel computing is described. This implementation is designed to handle large data sets such as those occurring in data mining with several millions of observations on several hundreds of variables. For such large data sets it is crucial to have a fast, parallel implementation for fitting generalised additive models to allow an exploratory analysis of the data within a reasonable time. The approach used divides the data into several blocks (groups) and fits a (generalised) additive model to each block. These models are then merged to a single, final model. It is shown that this approach is very efficient as it allows the algorithm to adapt to the structure of the parallel computer (number of processors and amount of internal memory).
Chunking was proposed for speedup purposes in @cite_1 . Here only a specific estimator was considered, a certain additive nonparametric regression model.