Adjustable Notch Filter help
In this line:


  anf->s.theta = -anf->p.gamma * anf->s.x1 * anf->s.theta * anf->s.err;


you seem to be conflating $\theta$ and $\dot{\theta}$ (the derivative).

Equation 5 of the paper is:

$$
\dot{\theta} = -\gamma x_1 \theta \left[ y(t) - \sum_{l=1}^n \dot{x}_l \right]
$$

Note that the left-hand term is the derivative of $\theta$, not $\theta$ itself.