Conditional bias-variance decomposition of MSE

The MSE can be decomposed as follows:

\begin{align*} \mathbb{E}\left[(\hat{\theta} - \theta)^2\right] &= \mathbb{E}\left[\left(\hat{\theta} - \mathbb{E}(\hat{\theta}) + \mathbb{E}(\hat{\theta}) - \theta\right)^2\right]\\ &= \mathbb{E}\left[\hat{\theta} - \mathbb{E}(\hat{\theta})\right]^2 + \mathbb{E}\left[\left(\mathbb{E}(\hat{\theta}) - \theta\right)^2\right] + 0\\ &= \mathbb{E}\left[\hat{\theta} - \mathbb{E}(\hat{\theta})\right]^2 + \left(\mathbb{E}(\hat{\theta}) - \theta\right)^2 \\ &= Var(\hat{\theta}) + Bias(\hat{\theta})^2 \end{align*}

Suppose $$X$$ is a random variable. Does a similar bias-variance decomposition exist for $$\mathbb{E}\left[\left.(\hat{\theta} - \theta)^2\right|X\right]$$?

\begin{align*} \mathbb{E}\left[\left.(\hat{\theta} - \theta)^2\right|X\right] &= \mathbb{E}\left[\left.\left(\hat{\theta} - \mathbb{E}(\hat{\theta}|X) + \mathbb{E}(\hat{\theta}|X) - \theta\right)^2\right|X\right]\\ &= \mathbb{E}\left[\left.\left(\hat{\theta}- \mathbb{E}(\hat{\theta}|X)\right)^2\right|X\right] + \mathbb{E}\left[\left.\left(\mathbb{E}(\hat{\theta}|X) - \theta\right)^2\right|X\right] + 2 \mathbb{E}\left[\left.\left(\hat{\theta} - \mathbb{E}(\hat{\theta}|X)\right)\left(\mathbb{E}(\hat{\theta}|X) - \theta\right)\right|X\right]\\ &= Var(\hat{\theta}|X) + \mathbb{E}\left[\left.\left(\mathbb{E}(\hat{\theta}|X) - \theta\right)^2\right|X\right] + 0\\ &= Var(\hat{\theta}|X) + \mathbb{E}\left[\left.\left(\mathbb{E}(\hat{\theta}|X) - \theta\right)^2\right|X\right] \end{align*}

Is the above correct?

Also, can the second term be written as the following?

\begin{align*}\mathbb{E}\left[\left.\left(\mathbb{E}(\hat{\theta}|X) - \theta\right)^2\right|X\right] &= \mathbb{E}\left[\left.\mathbb{E}(\hat{\theta}|X)^2 - 2\mathbb{E}(\hat{\theta}|X)\theta + \theta^2\right|X\right]\\ &= \mathbb{E}(\hat{\theta}|X)^2- 2\mathbb{E}(\hat{\theta}|X)\theta + \theta^2 \end{align*}

Putting it altogether, we have:

$$\mathbb{E}\left[\left.(\hat{\theta} - \theta)^2\right|X\right] = Var(\hat{\theta}|X)+\mathbb{E}(\hat{\theta}|X)^2- 2\mathbb{E}(\hat{\theta}|X)\theta + \theta^2$$

Edit:

By law of iterated expectation,

\begin{align*} \mathbb{E}\left[\left(\hat{\theta} - \theta\right)^2\right] &= \mathbb{E}\left[\mathbb{E}\left[\left.\left(\hat{\theta} - \theta\right)^2\right|X\right]\right]\\ &= \mathbb{E}\left[Var(\hat{\theta}|X)+\mathbb{E}(\hat{\theta}|X)^2- 2\mathbb{E}(\hat{\theta}|X)\theta + \theta^2\right]\\ &\overset{?}{=} Var(\hat{\theta}) + Bias(\hat{\theta})^2 \end{align*}

I'm not sure I see how the last line would hold?

Yes, it is correct. You can think of the original equation as a special case of the conditional one, where nothing is given. Your expansion of the second term is also correct. You can also sanity check the last equation by removing $$X$$'s as follows: $$\operatorname{Var}(\hat{\theta}|X)+\mathbb{E}(\hat{\theta}|X)^2- 2\mathbb{E}(\hat{\theta}|X)\theta + \theta^2\rightarrow \operatorname{Var}(\hat{\theta})+\underbrace{\mathbb{E}(\hat{\theta})^2- 2\mathbb{E}(\hat{\theta})\theta + \theta^2}_{\operatorname{Bias}(\hat\theta)^2}$$

• Thanks. I have a follow up question: I'm not sure how to verify that $\mathbb{E}\left[\left(\hat{\theta} - \theta\right)^2\right] = \mathbb{E}\left[\mathbb{E}\left[\left.\left(\hat{\theta} - \theta\right)^2\right|X\right]\right]$ holds in this case. I've edited my original post. Do you have any pointers? Commented Jan 17, 2021 at 17:45
• Normally you don't have to show that some other way because you've done it already above, however total variance formula probably helps. Commented Jan 17, 2021 at 18:01

Suppose the estimator $$\hat{\theta}$$ is a function of random variable $$X$$ and denoted as $$\hat{\theta} = g(X)$$, then when $$X=x$$, $$\hat{\theta} = g(x)$$, or a constant value.

The term $$\mathbb{E}(\hat{\theta}|X)$$ introduced in the bias-variance decomposition is a function of $$X$$ which takes value $$\mathbb{E}(\hat{\theta}=g(x)|X=x)=g(x)$$ when $$X=x$$. Because of this, the function $$\mathbb{E}(\hat{\theta}|X)$$ is the same function as $$\hat{\theta} = g(X)$$. Similarly, the term $$Var(\hat{\theta}|X)$$ as a function of $$X$$, takes value $$Var(\hat{\theta} = g(x) | X=x)=0$$ when $$X=x$$. So $$Var(\hat{\theta}|X)$$ has a constant value of 0.

Using these information, introducing the term $$\mathbb{E}(\hat{\theta}|X)$$ seems to lead the expansion of $$\mathbb{E}[(\hat{\theta}-\theta)^2|X]$$ back to its original form.

If we instead introduce the term $$\mathbb{E}(\theta|X)$$ such that $$\mathbb{E}[(\hat{\theta}-\theta)^2|X] = \mathbb{E}[(\hat{\theta}-\mathbb{E}(\theta|X)+\mathbb{E}(\theta|X)-\theta)^2|X] = ...$$ after expanding the terms, you should be able to get $$\mathbb{E}[(\hat{\theta}-\theta)^2|X] = \mathbb{E}[(\theta-\mathbb{E}(\theta|X))^2|X] + \mathbb{E}[(\mathbb{E}(\theta|X)-\hat{\theta})^2|X]$$. Note that the first term here is just the conditional variance of $$\theta$$. By setting the estimator $$\hat{\theta}=\mathbb{E}(\theta|X)$$, the second term becomes 0, and the mean squared error given $$X$$ is minimized.

This answer https://stats.stackexchange.com/q/164391 may provide some more interpretations of the second term in the result $$\mathbb{E}[(\mathbb{E}(\theta|X)-\hat{\theta})^2|X]$$ and how it relates to bias-variance decomposition. From what I've seen before, the bias-variance decomposition is usually done under the classical inference.
{"found_math": true, "script_math_tex": 0, "script_math_asciimath": 0, "math_annotations": 0, "math_alttext": 0, "mathml": 0, "mathjax_tag": 0, "mathjax_inline_tex": 1, "mathjax_display_tex": 0, "mathjax_asciimath": 0, "img_math": 0, "codecogs_latex": 0, "wp_latex": 0, "mimetex.cgi": 0, "/images/math/codecogs": 0, "mathtex.cgi": 0, "katex": 0, "math-container": 34, "wp-katex-eq": 0, "align": 0, "equation": 0, "x-ck12": 0, "texerror": 0}