The problem of statistical learning is to construct a predictor of a random variable Y as a function of a related random variable X on the basis of an i.i.d. training sample from the joint distribution of (X, Y). Allowable predictors are drawn from some specified class, and the goal is to approach asymptotically the performance (expected loss) of the best predictor in the class. We consider the setting in which one has perfect observation of the X-part of the sample, while the Y-part has to be communicated at some finite bit rate. The encoding of the Y-values is allowed to depend on the X-values. Under suitable regularity conditions on the admissible predictors, the underlying family of probability distributions and the loss function, we give an information-theoretic characterization of achievable predictor performance in terms of conditional distortion-rate functions. The ideas are illustrated on the example of nonparametric regression in Gaussian noise.
@cite_5: Some results from classical estimation theory are extended to the case in which data must be communicated from several places where observations are made to the place where the estimate is generated. Particular emphasis is placed on determining how the variance of an unbiased estimator depends on the communication rates. Explicit result are given for Gaussian sources. >
@cite_3: We analyze the following model: One person, called «helper» observes an outcome x n =(x 1 ,...,x n )∈#7B-X n of the sequence X n =(X 1 ,...,X n ) of i.i.d. RV's and the statistician gets a sample y n =(y 1 ,...,y n ) of the sequence Y n (θ,x n ) of RV's with a density Π t=1 n f(y t '(θ,x t ). The helper can give some (side) information about x n to the statistician via an encoding function s n : #7B-X n →N with rate (s n )=(1 n) log#range (s n )≤R. Based on the knowledge of s n (#7B-X n ) and y n the statistician tries to estimate θ by an estimator θ n . For a maximal mean square error we establish a Cramer-Rao type bound and, in case of a finite #7B-X, prove asymptotic achievability of this bound under certain conditions
@cite_2: This paper presents a survey of the literature on the information-theoretic problems of statistical inference under multiterminal data compression with rate constraints. Significant emphasis is put on problems: (1) multiterminal hypothesis testing, (2) multiterminal parameter estimation and (3) multiterminal pattern classification, in either case of positive rates or zero rates. In addition, the paper includes three new results, i.e., the converse theorems for all problems of multiterminal hypothesis testing, multiterminal parameter estimation, and multiterminal pattern classification at the zero rate.
Previously, the problem of statistical estimation from compressed data was considered by Zhang and Berger @cite_5 , Ahlswede and Burnashev @cite_3 and Han and Amari @cite_2 from the viewpoint of multiterminal information theory. In these papers, the underlying family of distributions of @math is parametric, i.e., of the form @math , where @math is a subset of @math for some finite @math , and one wishes to estimate the true" parameter @math . The i.i.d. observations @math are drawn from @math , and the input part @math is communicated to the statistician at some rate @math , while the output part @math is communicated at some rate @math . The present work generalizes to the nonparametric setting the case considered by Ahlswede and Burnashev @cite_3 , namely when @math . To the best of the author's knowledge, this paper is the first to consider the problem of nonparametric learning from compressed observations with side information.