How does training for DQN work if messing up in the environment in costly? Suppose that we want to train a car to drive in the real world and decide to use Reinforcement Learning (specifically, DQN) for that. I am a bit confused about how training generally works.
Is it that we are exploring the environment at the same time that we are training the Q network? If so, is there not a way to train the Q network before actually going out into the real world? And then, aren't there millions of possible states in the real world? So, how does RL or I guess the neural network generalize so that it can function during rush hour, empty roads, etc.
In general, you need to actively explore the environment to gather data to train your Q network. However, especially in your self-driving car example, you might be looking for Batch RL. In Batch RL you start with a given, fixed dataset of transitions (state, action, reward, next state) and you learn a policy (or Q function) based on the dataset without exploring. The Batch-Constrained Q-learning algorithm (BCQ) is a good example of this.
With regards to acquiring the data necessary for Batch RL, that's another story. I suppose one could collect data through sensors in a car, and label datapoints with rewards according to some reward function.