How to train a logical XOR with reinforcement learning? After reading an excellent BLOG post Deep Reinforcement Learning: Pong from Pixels and playing with the code a little, I've tried to do something simple: use the same code to train a logical XOR gate.

But no matter how I've tuned hyperparameters, the reinforced version does not converge (gets stuck around -10). What am I doing wrong? Isn't it possible to use Policy Gradients, in this case, for some reason?

The setup is simple:


3 inputs (1 for bias=1, x, and y), 3 neurons in the hidden layer and 1 output.
The game is passing all 4 combinations of x,y to the RNN step-by-step, and after 4 steps giving a reward of +1 if all 4 answers were correct, and -1 if at least one was wrong.
The episode is 20 games


The code (forked from original and with minimal modifications) is here: https://gist.github.com/Dimagog/de9d2b2489f377eba6aa8da141f09bc2

P.S. Almost the same code trains XOR gate with supervised learning in no time (2 sec).
Reinforcement learning is used when we know the outcome we want, but not how to get there which is why you won't see a lot of people using it for classification (because we already know the optimal policy, which is just to output the class label). You knew that already, just getting it out of the way for future readers!

As you say, your policy model is fine - a fully connected model that is just deep enough to learn XOR.  I think the reward gradient is a little shallow - when I give a reward of +1 for "3 out 4" correct and +2 for "4 out of 4", then convergence happens (but very slowly).