IMU Madgwick filter pitch effect on roll and yaw
From the figure it looks like the yaw and roll angles become significant only when the pitch angle come close to +-90 deg. If pitch angle is the middle angle in the Euler angle representation for your algorithm, perhaps your algorithm calculations are running into a singularity.
Wikipedia - Euler angles
Wikipedia - gimbal lock.
Solution

Choose a different algorithm or
A different representation of attitude other than Euler angles or
Don't operate your platform (aircraft / quad copter / other) at high pitch angles.

Edit
OP has commented

Algorithm works in quaternion base in its math. Euler only represantation, after getting quaternion. So Euler post-processed.

The conversion from quaternion to Euler also has singularity / gimbal lock. The conversion equation taken from Wikipedia is
$$
\begin{bmatrix}\phi \\\theta \\\psi \end{bmatrix}
=
\begin{bmatrix}
\arctan{\frac {2(q_{0}q_{1}+q_{2}q_{3})}{1-2(q_{1}^{2}+q_{2}^{2})}}
\\
\arcsin(2(q_{0}q_{2}-q_{3}q_{1}))
\\
\arctan{\frac {2(q_{0}q_{3}+q_{1}q_{2})}{1-2(q_{2}^{2}+q_{3}^{2})}}\end{bmatrix}
$$
At pitch angle \$\rightarrow 90\$ deg,  \$q_2 \rightarrow sin(90/2)\cdot1 = \frac{1}{\sqrt2}\$.
The denominator in the calculation of \$\psi\$ becomes
$$
1-2(q_{2}^{2}+q_{3}^{2}) = 1 - 2 (\frac{1}{\sqrt2^2} + 0^2) = 0
$$
Division by zero results. Using \$\arctan2\$ may stop any software error from occurring, but it doesn't solve the illposedness of the underlying problem.