Snake game: snake converges to going in the same direction every time This is a q-learning snake using a neural network as a q function aproximator and I'm losing my mind here the current model it's worst than the initial one.

The current model uses a 32x32x32 MLPRegressor from scikit-learn using relu as activation function and the adam solver.

The reward function is like following:


death reward = -100.0
alive reward = -10.0
apple reward = 100.0


The features extracted from each state are the following:


what is in front of the snake's head(apple, empty, snake)
what is in the left of the snake's head
what is in the right of the snake's head
euclidian distance between head and apple
the direction from head to the apple measured in radians
length of the snake


One episode consists of the snake playing until it dies, I'm also using in training a probability epsilon that represent the probability that the snake will take a random action if this isn't satisfied the snake will take the action for which the neural network gives the biggest score, this epsilon probability gradually decrements after each iteration.

The episode is learned by the regressor in reverse order one statet-action at a time.

However the neural network fails too aproximate the q function, no matter how many iterations the snake takes the same action for any state.

Things I tried:


changing the structure of the neural network
changing the reward function
changing the features extracted, I even tried passing the whole map to the network


Code (python): https://pastebin.com/57qLbjQZ
There are two problems here.


The code you posted doesn't incrimentally train your multilayer perceptron. Instead, it effectively re-randomizes the weights, and then re-fits the model each time you call .fit() at lines 35 &amp; 54. Using SKLearn's _fit() function with Incremental=true might solve this, or you can package up the data into a larger batch, and train on that offline instead after several episodes.
Your reward function makes it painful to be alive, and doesn't give enough benefits through the Apples to make up for this. There are 100 squares that could contain the apple. On average, the apple will spawn about 5 squares from the snake in each direction. Since the snake can't move diagonally, that's 10 moves (5 left/right, 5 up/down). That means that if the snake plays perfectly, then on average, it might be able to get zero reward total. In practice, the snake will not play perfectly. This means living gives negative expected reward.


In contrast, if the snake can kill itself, it will stop getting negative rewards. The reward function you've used is maximized by getting big enough to run into your own tail as fast as possible. The snake should be able to do this after eating 3 apples I think. There is some incentive to hunt for food well, but not much compared with hitting your own tail as soon as possible.

If you want the snake to learn to hunt for the food, reduce the penalty for being alive to -1, or even -0.1. The snake will be much more responsive to signals from the food.