Transform coding gain of two channel Haar Wavelet transform (first stage)
I am not sure why you're trying to do this in MATLAB. This can be done quite easily with pen and papers.



Another way of seeing what you're doing here is the following:


Build vectors of size 2 from the sequence $X$
$$[X(0),~X(1)], [X(2),~X(3)], [X(4),~X(5)]\dots$$
Apply an orthogonal transform on each vector to obtain the corresponding transform coefficients $Y$.


The order 2 orthogonal transform you're using can mathematically be written as
$$\begin{bmatrix}Y_0 \\ Y_1\end{bmatrix} = \frac{1}{\sqrt{2}}\begin{bmatrix}1 &amp; 1 \\ 1 &amp; -1\end{bmatrix}\begin{bmatrix}X(n) \\ X(n+1)\end{bmatrix}$$
where $Y_0$ and $Y_1$ are the transform coefficients and $X(n)$ and $X(n+1)$ are successive (hence correlated) samples from the process to be compressed. From this, you can easily compute $\sigma^2_0$ (the variance of $Y_0$) and $\sigma^2_1$ (the variance of $Y_1$). Indeed, because $X$ is zero-mean (and so are the transform coefficients), you know that
$$\sigma^2_i = R_{Y_i}(0) = E\{Y_iY^*_i\}$$
for $i = 1, 2$.
Then, you simply need to plug in the expression of $Y_i$ as a function of $X(n)$ and $X(n+1)$ to find $\sigma^2_i$ as a function of $R_X(0)$ and $R_X(\pm1)$. At the end, you should obtain
$$\sigma^2_0 = R_X(0) + R_X(1) = 19.3,$$
$$\sigma^2_1 = R_X(0) - R_X(1) = 0.7.$$
You can clearly see from this the energy compaction in the first transform coefficient, which will contributes to the coding gain. You now have everything you need to compute the coding gain (note that $N$ in your expression for the coding gain is the number of transform coefficients, 2 in this case).