What is the purpose of a Neural Network in Reinforcement Learning when we have a Q-learning update rule? I'm confused as to the purpose of training a neural network (NN) for reinforcement learning (RL) tasks such as Gridworld. In RL tasks, namely q-learning, we have a q-learning update rule, which is designed to take some state and action and compute the value of that state-action pair.
Performing this process several times will eventually produce a table of states and what action will likely lead to a high reward.
In RL examples, I've seen them train a neural network to output q-values and a loss function like MSE to compute the loss between the q-learning update rule q-value and the NN's q value.
So:
(a) Q-learning update rule-&gt; outputs target Q-values
(b) NN -&gt; outputs Q values
MSE to compute the loss between (a) and (b)
So, given we already know what the target Q-value is from a, why do we need to train a NN?
I don't think people generally do use neural nets for grid world. As long as the state and action spaces are small enough, you should be able to store Q values in a table like you suggested. Neural nets come in handy when the state space is very large (or even continuous), so you can't afford to store a table of Q values. Also, neural nets have the ability to generalize across &quot;similar&quot; states -- for instance, if two states are very similar the neural net would likely produce similar values for those states, whereas a tabular implementation might not have seen enough data to accurately estimate the Q values of both.