Let $X, Y$ be two random variables, with $X$ taking values in $\Bbb R^n$ and $Y$ taking values in $\Bbb R$.

Then we can look at the function $h: \Bbb R^n \to \Bbb R$ given by $$\beta \mapsto \Bbb E[(Y-X^T\beta)^2]$$ It is claimed that the gradient of $h$ is given by $$\nabla h = \Bbb E[2X(X^T\beta-Y)]$$

This seems like a special case of the identity 

> 
> $$\nabla \Bbb E[f]=\Bbb E [\nabla f]$$
> 
> 
> 

Where the expectation is taken over the mutual distribution of some random variables. 

Formally, We want the following: Suppose $X\_1,...,X\_m$ are random variables returning values in some sets $A\_i$ with some given mutual probability distribution. Then for every function $f: \Bbb R^n \times \prod A\_i \to \Bbb R$, for every $\beta \in \Bbb R^n$ we can form the random variable $f(\beta, X\_1,...,X\_m)$ and take its expectation. Taking different values of $\beta$ gives rise to a function $\Bbb R^n \to \Bbb R$. We claim that its gradient is equal to the vector obtained by first fixing the values of $X\_1,...,X\_m$ and taking the gradient of the resulting function $\Bbb R^n \to \Bbb R$, and this gives a random variable returning values in $\Bbb R^n$, for which we can take the expectation.