The magnitude spectrum of a sharpening filter
UPD: I've figured it out myself. I was inattentive with the exponentials while deriving the expression for the low-pass filter $H(u,v)$. There shouldn't be $\cos(6(\mu+\nu))$ term, then the result is fine and $$\Re \big\{H(u,v)\big\}=\frac{1}{25} \cdot \frac{\sin\frac{5\mu}{2}\sin\frac{5\nu}{2}}{\sin\frac{\mu}{2}\sin\frac{\nu}{2}}$$
Therefore,
\begin{align*}
|G(u,v)|^2 &amp;= \big[(1+\lambda)-\lambda H(u,v)\big] \big[(1+\lambda)-\lambda H^*(u,v)\big] \\
&amp; = (1+\lambda)^2-(1+\lambda)\lambda\cdot 2\Re \big\{H(u,v)\big\}+\lambda^2|H(u,v)|^2 = [1+\lambda - \lambda H(u,v)]^2
\end{align*}
And the result is exactly what I was expecting to get.