Condition for optimal value to be the average
You don't need to assume anything about the data in order for the average to be optimal (in a least squares sense) for certain estimation problems. As an example, assume you want to estimate a constant value $x$ and you are given $N$ noisy measurements:

$$y_n=x+e_n,\quad n=0,1,\ldots,N-1\tag{1}$$

where $e_n$ is the error of the $n^{th}$ measurement. If our goal is to minimize the mean squared error

$$\frac{1}{N}\sum_{n=0}^{N-1}e_n^2=\frac{1}{N}\sum_{n=0}^{N-1}(y_n-x)^2\tag{2}$$

then the optimal estimate $\hat{x}$ is obtained by minimizing (2). Taking the derivative of the right-hand side of (2) with respect to $x$ and setting it to zero gives

$$\frac{2}{N}\sum_{n=0}^{N-1}(y_n-\hat{x})=0\tag{3}$$

From (3) the optimum estimate is obtained as

$$\hat{x}=\frac{1}{N}\sum_{n=0}^{N-1}y_n\tag{4}$$

which is simply the average of all observations.