How can I improve this RL algorithm?
There's no reason to think that reinforcement learning would be effective here.  Indeed, the surprise would be if it did work!  Machine learning isn't a magic silver bullet.  I wouldn't expect it to be of any assistance here; the problem relates to deep questions about the structure of the general linear group, which might be quite challenging for your policy network to learn.

Your problem is known as the membership problem for the general linear group, which might be hard (i.e., no efficient algorithm is known), depending on which field you are working over.  See https://cstheory.stackexchange.com/q/38000/5038.  If we take $T=0$ and are working with $n \times n$ matrices, this is called the matrix mortality problem; it is undecidable for $n \ge 3$.  See also https://en.wikipedia.org/wiki/List_of_undecidable_problems#Problems_about_matrices.  So, in the general case, your problem is undecidable.

Obviously, you shouldn't expect your approach to successfully solve an undecidable problem.