Title: Scalable and Efficient Statistical Inference with Estimating Functions in the MapReduce Paradigm for Big Data,
Abstract: The theory of statistical inference along with the strategy of
divide-and-conquer for large- scale data analysis has recently attracted
considerable interest due to great popularity of the MapReduce programming
paradigm in the Apache Hadoop software framework. The central analytic task in
the development of statistical inference in the MapReduce paradigm pertains to
the method of combining results yielded from separately mapped data batches.
One seminal solution based on the confidence distribution has recently been
established in the setting of maximum likelihood estimation in the literature.
This paper concerns a more general inferential methodology based on estimating
functions, termed as the Rao-type confidence distribution, of which the maximum
likelihood is a special case. This generalization provides a unified framework
of statistical inference that allows regression analyses of massive data sets
of important types in a parallel and scalable fashion via a distributed file
system, including longitudinal data analysis, survival data analysis, and
quantile regression, which cannot be handled using the maximum likelihood
method. This paper investigates four important properties of the proposed
method: computational scalability, statistical optimality, methodological
generality, and operational robustness. In particular, the proposed method is
shown to be closely connected to Hansen's generalized method of moments (GMM)
and Crowder's optimality. An interesting theoretical finding is that the
asymptotic efficiency of the proposed Rao-type confidence distribution
estimator is always greater or equal to the estimator obtained by processing
the full data once. All these properties of the proposed method are illustrated
via numerical examples in both simulation studies and real-world data analyses.