What is the purpose of a Neural Network in Reinforcement Learning when we have a Q-learning update rule?
I don't think people generally do use neural nets for grid world. As long as the state and action spaces are small enough, you should be able to store Q values in a table like you suggested. Neural nets come in handy when the state space is very large (or even continuous), so you can't afford to store a table of Q values. Also, neural nets have the ability to generalize across &quot;similar&quot; states -- for instance, if two states are very similar the neural net would likely produce similar values for those states, whereas a tabular implementation might not have seen enough data to accurately estimate the Q values of both.