Lets say I have an operator given by:
$$\hat{A}|n\rangle=|n+1\rangle$$ where $n$=integer

and another operator given by:
$$\hat{B}|n\rangle=|n-1\rangle$$

Now let's suppose for my case states $|n\rangle$ where $n$ > 2 are forbidden (which means we need to put it zero).

Now which one of these are correct $$\hat{B}\hat{A}|2\rangle=\hat{B}|3\rangle=\hat{B}\*0=0$$

or

$$\hat{B}\hat{A}|2\rangle=\hat{B}|3\rangle=|2\rangle$$

Edit: Adding information about my system after suggestion in a comment

The system is a cavity filled with a laser input that drives a mechanical oscillator. At the low temperature and driving conditions, we consider a basis state $|m,n\rangle$ with $m,n$ are photon and phonon numbers respectively and truncate it according to the condition $m+n< 3$ as:
$$|\Psi\rangle=C\_{00}|0,0\rangle+C\_{01}|0,1\rangle+C\_{10}|1,0\rangle ......C\_{03}|0,3\rangle+C\_{30}|3,0\rangle$$

Now my Hamiltonian has in an operator term of the form $\hat{B}\hat{A}$ for the photon such that when it gets applied to the $C\_{30}|3,0\rangle$ term of
the above wavefunction it gives:

$$\hat{B}\hat{A}|3,0\rangle=\hat{B}|4,0\rangle$$

What will this give?

Can I now operate $\hat{B}$ on this "forbidden" state to get back the allowed $|3,0\rangle$ state or should directly put it equal to zero since the operator is trying to operate on a forbidden state?

I have gotten two contradictory answers and adopting one from the other causes a difference in the answer. Which one is correct or rather which is more accurate?