Designing appropriate filter by choosing values for $A$ and $B$
Bottom Line:
$$A &lt;1$$
$$B =2\cos(\omega_n)$$
Where $\omega_n$ is the normalized angular frequency of the desired notch location (in this case for the OP with a sampling rate of 8KHz and notch at 400 Hz this would be $\omega_n = 2\pi400/8000 = \pi/10$), resulting in $B \approx 1.902$ and $A$ is the frequency notch bandwidth parameter; the closer $A$ is to 1 the tighter the bandwidth of the notch.
Details:
The straight-forward approach to eliminating narrow band noise at 400 Hz would be a classic 2nd order notch filter. This is detailed at this post:
Transfer function of second order notch filter
This approach is done by simply placing a zero at the frequency of interest, and by adding a pole close to the zero, but inside the unit circle of course to be stable, we can adjust the bandwidth of the notch by the proximity of this pole to that zero, meaning $|p|&lt;1$, but close to 1.
Using real coefficients, meaning complex conjugate zeros and poles as detailed in the linked post, this results in the transfer function:
$$ H(z) = \frac{1+a}{2}\frac{z^2-2z\cos(\omega_n)+1}{(z^2-2az\cos(\omega_n)+a^2)} $$
Which as a difference equation is exactly in the form that the OP has used and therefore given this is the same filter, answers the OP's question. This is is derived from the z-transform as follows (we can ignore the gain scaling parameter $(1+a)/2$ which for most cases is close to 1, and express in decreasing powers of z (and assuming I didn't make simple algebra errors!):
$$ \frac{Y(z)}{X(z)} = \frac{1-2z^{-1}\cos(\omega_n)+z^{-2}}{(1-2az^{-1}\cos(\omega_n)+a^2z^{-2})} $$
$$ Y(z) (1-2az^{-1}\cos(\omega_n)+a^2z^{-2})= X(z)(1-2z^{-1}\cos(\omega_n)+z^{-2})$$
$$Y(z)- Y(z)2a\cos(\omega_n)z^{-1}+Y(z)a^2z^{-2} = X(z)-X(z)2\cos(\omega_n) z^{-1} +X(z)z^{-2} $$
And then with the inverse Z transform we get the desired coefficients that the OP wanted:
$$y[n] - 2a\cos(\omega_n)y[n-1] + a^2y[n-2] = x[n]-2\cos(\omega_n)x[n-1] + x[n-2] $$
With comparison to OP's expression:
$$y[n] - ABy[n-1] +A^2y[n-2] = x[n] - Bx[n-1] +x[n-2]$$
Results in the following:
$$A = a$$
$$B =2\cos(\omega_n)$$
Where $\omega_n$ is the normalized angular frequency of the desired notch (for the OP's sampling rate and notch frequency this would be $\omega_n = 2\pi400/8000 = \pi/10$) and $a$ is the notch bandwidth parameter ($a&lt;1$, the closer a is to 1 the tighter the notch, refer to the linked post for complete details).