Compress-and-Estimate Source Coding for a Vector Gaussian Source

We consider the remote vector source coding problem in which a vector Gaussian source is to be estimated from noisy linear measurements. For this problem, we derive the performance of the compress-and-estimate (CE) coding scheme and compare it to the optimal performance. In the CE coding scheme, the remote encoder compresses the noisy source observations so as to minimize the local distortion measure, independent from the joint distribution between the source and the observations. In reconstruction, the decoder estimates the original source realization from the lossy-compressed noisy observations. For the CE coding in the Gaussian vector case, we show that, if the code rate is less than a threshold, then the CE coding scheme attains the same performance as the optimal coding scheme. We also introduce lower and upper bounds for the performance gap above this threshold. In addition, an example with two observations and two sources is studied to illustrate the behavior of the performance gap.


I. INTRODUCTION
The Distortion-Rate Function (DRF) describes the minimum attainable average distortion when recovering an information source compressed to within a target bit rate for asymptotically large blocklength. The setting in which the source sequence is not directly available at the encoder and only noisy observations are provided is referred to as the remote or indirect source coding setting [1,Sec. 3.5]. In general, the optimal coding scheme in the indirect source coding setting depends on the joint statistics of the underlying source and the noisy observations. Therefore, the optimal scheme is not applicable when the encoder is designed without knowledge of this statistics. Namely, the encoder is either unaware of the existence of the underlying source and regards the observed process as the target for compression, or utilizes a general-purpose compression strategy that does not adapt to the underlying source. Under these limitations, the encoder will compress its sequence of observations based on a distortion criterion defined only with respect to this observed sequence. The decoder, having full knowledge of the joint statistics of the underlying source and its observations, estimates the source from the output of the encoder. This compress-and-estimate (CE) scheme was proposed and studied in [2]. In this paper we extend the results of [2] to the Gaussian vector remote source coding problem, thus providing a better understanding of the loss of performance caused by partial system knowledge at the remote encoders.
Previous Work: The indirect source coding problem was first considered in [3], where it was shown that the optimal trade-off between code-rate and distortion is characterized by a single-letter expression, denoted as the indirect DRF (iDRF). Moreover, it was proven in [1], [4] that the iDRF can be achieved by first estimating the remote source from the input to the observer, and then encoding this estimate optimally with respect to the code-rate constraint. Therefore, the optimal indirect source coding scheme can be referred to as the estimate-and-compress scheme. The CE scheme for the indirect source coding setting was introduced in [5] and was motivated by the difficulty in implementing an estimation step before encoding. A single-letter expression characterizing the minimal distortion in the CE setting with multiple encoders was derived in [2]. Moreover, it was shown in [2] that this expression has a closed form expression in the quadratic Gaussian setting of a scalar Gaussian source observed through multiple linear measurements corrupted by Gaussian noise. The optimal rate-allocation across the multiple encoders was considered in [6].
Other works that consider communication with non-optimal encoding due to missing source statistics or codebook information include the oblivious processing channel coding problem of [7], the minimax source coding of [8], and the mismatched encoding problem of [9]. In particular, as explained in this paper, the CE scheme can be seen as a special case of the mismatched encoding problem.
Contributions: In this work, we consider the CE scheme in the quadratic Gaussian setting with a vector Gaussian source. Namely, we consider the minimal distortion in estimating a vector Gaussian source from an encoded version of its noisy measurement, where the encoder employs a source code that is optimal with respect to the noisy measurements under a quadratic distortion. For this setting, we show that the CE scheme is optimal when the code-rate is below a certain threshold that depends on the spectrum of the observed vector. That is, when the code-rate is below this threshold, lack of underlying source statistics causes no penalty in the distortion. Above this threshold, we derive upper and lower bounds on the distortion difference between the CE scheme and the optimal scheme using an inequality between the arithmetic mean and the geometric mean (AM-GM inequality). In addition, we consider an example with two sources and two observations to illustrate the behavior of performance gap. Paper Organization: The remainder of the paper is organized as follows: in Sec II we introduce the problem formulation. The main results are presented in Sec. III. Sec. IV concludes the paper.
Throughout, for the sake of brevity, we only provide a sketch of the proofs in the body text. The complete proofs are available in the appendices.
Notation: We denote [n] = {1, . . . , n} for n ∈ N, and a + = max{a, 0}, for a ∈ R. Also, we define log + (x) as log(max{1, x}). With I L we indicate the identity matrix of size L × L. With diag(v) we indicate the matrix with the elements of v on the diagonal. For a square matrix C, denote by λ l (C) the l th largest eigenvalue. For C positive semi-definite and for k ∈ [rank(C)], we define functions R k (C) and θ k (C, R) as follows: and II. SYSTEM MODEL We consider the indirect source coding setting in Fig. 1 in which an M -dimensional Gaussian source is observed through a vector additive white Gaussian noise (AWGN) channel at a remote encoder. The random source sequence X n = (X n 1 . . . X n M ) is obtained through n i.i.d. draws from the jointly Gaussian distribution with zero mean and covariance matrix I. The remote encoder obtains the noisy observation vector for some A ∈ R L×M , assumed full rank, with noise Z ∼ N (0, σ 2 I L ) ∈ R L . The encoder produces the index m ∈ 1 . . . 2 nR to encode its observation and the index m is noiselessly communicated to a central processing unit that produces the reconstruction sequence X n (m). Given a value R, we wish to determine the minimum average quadratic distortion between the original source sequence X n and its reconstruction X n , normalized over the source dimension M , which is: where the expectation is taken with respect to all source and channel realizations.
Remark II.1. The model in (3) in which X has a general covariance matrix Σ X can be reduced to the case Σ X = I without loss of generality. This is obtained by letting In the following, we assume that the matrix A in (3) is full rank and define r = min{M, L}. We use {λ l , l ∈ [L]} to denote the eigenvalues of AA T , sorted in descending order. Note that the covariance matrix of Y, i.e. Σ Y , and the conditional covariance matrix of X|Y, i.e. Σ X|Y , are respectively obtained as Following (5) where the infimum is taken over all joint probability distributions of Y and X such that the per letter mutual information I(Y; X) does not exceed R.
The iDRF for the model in (3) can be rewritten as where mmse(X|Y) is the minimum mean square error (MMSE) when estimating X from Y, i.e.
The solution of the optimization in (6) through the formulation in (7) results in the classic water-filling assignment of the compression rate for each observation. This solution yields a simple expression for the inverse of the iDRF, which is the rate-distortion function where where θ is chosen so that l D l = D. The expression in (10) shows that, in the optimal compression scheme, (10). In other words, the MMSE error in estimating each source component is controlled by the solution of the water-filling problem so that if the i th eigenvalue is below the water level, no rate is assigned to the compression of the estimate of the i th observation.
Compress-and-Estimate Vector Source Coding: The CE setting [5] considers the remote source coding problem in which each remote encoder compresses its noisy observation sequence so as to minimize a local distortion measure that depends only on the distribution of its observed sequence, and is otherwise independent from the distribution of the underlying source. The CE-DRF is the single letter expression of the distortion that can be attained by the CE coding scheme for the case of an i.i.d. source observed through a memoryless channel. Given a local distortion measure d l and a probability distribution P * Y,Y that satisfies I(Y; Y) = R and Ed(Y l , Y l ) = D(R), where D(R) is the classic distortion-rate function with respect to compressing Y under distortion measure d l , the CE-DRF is defined as where the infimum is over all estimators of X given the noisy reconstructions Y. For the observation model in (3), the quadratic distortion in (4) at the central unit and as a local distortion at the remote encoder, the CE-DRF is expressed as where the joint distribution P Y, Y is described by the backward Gaussian channel Y = Y + UZ, where U is an orthogonal matrix that diagonalizes Σ Y , and the noise Z ∼ N (0,

III. MAIN RESULT
In the following, we derive conditions under which the CE-DRF in the vector quadratic Gaussian setting equals the iDRF. In addition, we derive bounds, both upper and lower, to the performance gap between the CE-DRF and the iDRF. We also provide a two-dimensional example and characterize the performance gap in different regions of R.

A. Conditions for Equality
We begin by deriving the conditions under which the CE-DRF and the iDRF coincide. To do so, we re-write the iDRF as the inverse of the rate-distortion function in (9).
In Prop. III.1, we express D X|Y (R) as a piecewise function of R over the intervals In each one of those intervals, the water-filling solution prescribes for k eigendirections of Σ X|Y to be compressed at the remote encoder. In other words, k in (14) corresponds to the number of D l > θ in (9).
We then obtain an expression of the CE-DRF.
Proposition III.2. When the rate R satisfies the CE-DRF in (12) is obtained as By comparing the expressions in Prop. III.1 and Prop. III.2 we note that, for R ≤ min{R 2 (Σ Y ), R 2 (Σ X|Y )}, the expressions in (14) and (16) are equal. This condition for equality can be generalized as in the following proposition.
The result in Prof. III.3 generalizes a previous result in [5] where equality was shown for the case of L = M = 1. Note that if r 0 = L = M , then we have D CE (R) = D X|Y (R) for any R. The functions D X|Y (R) and D CE (R) are continuous decreasing functions of R. We note that D CE (R) is only smooth between each pair of two rate thresholds, that is in the intervals . Each rate threshold R k (Σ Y ) determines a change in the slope of D CE (R). Despite the fundamental difference between these two functions, in the low rate regime of Prop. III.3 the two functions actually coincide.
Discussion: Note that both the CE and the optimal compression schemes suffer from a distortion mmse(X|Y), which follows from the fact that Y, instead of X, is observed. Moreover, the forward channel between the observation and its reconstruction for both transmission schemes corresponds to a vector additive Gaussian noise channel. In the optimal compression scheme, the noise is added along the eigenvectors of the covariance matrix Σ X|Y , while in the CE scheme the noise follows the direction of the eigenvectors of the matrix Σ Y . This distinction implies that the CE scheme might allocate compression resources to eigendirections which are not useful in estimating the underlying source X. As an example, assume that there exists λ l = 0. Recall (5a), when the l th eigenvalue of AΣ X A T is zero, the l th eigenvalue of Σ Y equals σ 2 , which solely comes from the covariance matrix of noise σ 2 I, and the l th eigenvalue of Σ X|Y satisfies λ l (Σ X|Y ) = λ l /(λ l + σ 2 ) = 0. Hence, the l th components in both optimal and CE settings are pure noise and contain no information of the source. For the optimal scheme, we see that the compression rate allocated for this component is always zero. In other words, the optimal scheme never activates a component that does not contain source information. However, we note that the CE scheme might activate components that are pure noise when R is sufficiently large. This is due to the fact that, in the optimal scheme, the encoder knows the joint statistics and can therefore avoid wasting rate resources on useless observations. In contrast, the encoder in the CE setting cannot recognize a pure noise component since it lacks knowledge of the source. On the other hand, when the rate is sufficiently small, only the largest eigendirection is actively compressed, in which case, perhaps surprisingly, the CE performance equals the optimal performance.

B. Performance Gap
We next upper bound the performance gap between the CE and the optimal performance, defined as for the regimes in which the equality conditions of Prop. III.3 does not hold.
Theorem III.4. The difference between the CE-DRF and the iDRF is bounded as Proof: Only a sketch of the proof is presented here. As an example, assume that there exists an R for which (13) and (15) hold for the same k. As argued above, this implies that the two schemes actively compress the same number of components. For this value of R we have where (20) follows from the classic AM-GM inequality. Through a similar reasoning, the inequality in (19) is obtained by exploiting a reverse AM-GM inequality in [10] to bound the largest difference between the CE and the optimal performance.
Th. III.4 shows that the difference between the CE-DRF and the optimal distortion is upper bounded by a function that decreases exponentially with R/L. To complement the result in Th. III.4, we introduce the following lower bound to the performance gap between the CE and the optimal performance.
Theorem III.5. For R > R 2 (Σ Y ), the difference between CE-DRF and iDRF is lower bounded as Proof: We provide a sketch of proof here. Suppose k satisfies (15). We have The first inequality follows from the fact that k satisfies (15) but not necessarily (13), and hence could be a non-optimal choice for the setting with full knowledge. The second inequality follows from a lower bound on the difference between AM and GM in [11,Sec. II]. Finally, (21) follows by noting the bound on R given in (15) and the fact that λ l 's are in descending order.
Recall that the performance gap can be zero for small R, the lower bound (21) is valid for The results in Th. III.4 and Th. III.5 show that the gap between the CE and the optimal performance decays exponentially in the rate-per-observation, R/L. The upper and lower bounds are always monotonically decreasing but, from numerical observations, they appear to be loose in the region for small R. Nonetheless they correctly capture the asymptotic decrease in the gap between the performance of the two schemes.

C. Two Observations, Two Sources Example
In this section we consider the case with two sources and two observations, i.e. M = L = 2. For brevity, we consider only the scenario Recall that, by the definition in (1), Following the assumption in (23). Given the considerations above, we derive the performance gap in three regions: (i) R ∈ (0, R 2 (Σ X|Y )], (ii) R ∈ (R 2 (Σ X|Y ), R 2 (Σ Y )], and (iii) R ∈ (R 2 (Σ Y ), ∞).
• R ∈ (R 2 (Σ X|Y ), R 2 (Σ Y )]: Here we have (25) In this region, the performance gap increases with R, and the maximal gap is which is achieved at In this case we have We see that in this region, the performance gap decays exponentially in 2R/L = R, which corresponds to the behavior predicted by Th. III.4 and Th. III.5.

IV. CONCLUSION
We have derived the performance of compress-and-estimate (CE) coding for a Gaussian vector, observed at the remote encoder through linear observations and further corrupted by additive white Gaussian noise. In the CE coding scheme, the remote encoder compresses its observation according to a local distortion measure which depends solely on the observation distribution and rate-per-symbol constraint. An estimator receives the encoded observations and uses them to estimate the remote source sequence. For this setting, we showed that when the rate is smaller than a threshold, the CE setting attains the optimal source coding performance. This threshold is obtained as a rather straightforward function of the eigenvalues of the observation matrix. Since the operation at the remote encoder depends only on the distribution of the observation, this result shows instances in which the optimal coding performance can be attained without the full system knowledge at the remote encoder. In addition, we derived upper and lower bounds on the performance gap between the CE scheme and the optimal scheme where the encoder has full knowledge of the underlying source statistics, which shows that the decay in the performance loss is exponential in the rate-per-observation in the region where the rate-per-symbol is large. Finally, for the case of two observations and two sources, a complete characterization of the behavior of the performance gap is derived.