How to train a logical XOR with reinforcement learning?
Reinforcement learning is used when we know the outcome we want, but not how to get there which is why you won't see a lot of people using it for classification (because we already know the optimal policy, which is just to output the class label). You knew that already, just getting it out of the way for future readers!

As you say, your policy model is fine - a fully connected model that is just deep enough to learn XOR.  I think the reward gradient is a little shallow - when I give a reward of +1 for "3 out 4" correct and +2 for "4 out of 4", then convergence happens (but very slowly).