Derivation of Circular Mean Square Error
The first term in $e^2_p$ is just the variation of the estimates around the true value.

The second term is due to the problem that happens at the end of the periodic region. Sometimes, the noise is enough to move the estimate from $-\pi+\alpha$ to $\pi - \beta$.

For example, the orange plot in the figure below represents the variation of the estimates for a true value around zero.  This variation is captured by the first term.

The blue plot in the figure below represents the variation of the estimates for the true value around the end of the period.  The trouble is that some estimates end up all the way at the other end of the periodic region.  This causes a bias to appear.  The second term in (36) is aimed at capturing that bias.