Is the PyTorch official tutorial really about Q-learning?
TL;DR: It is Q learning. However Q learning is basically sample-based value iteration, so not surprising you see a similarity.

Q learning* and value iteration are very strongly related. When considering action values, both approaches use the same Bellman equation for optimal policy, $q^*(s,a) = \sum_{r,s'}p(r,s'|s,a)(r+\gamma \text{max}_{a'} q^*(s', a'))$ as the basis for update steps. The differences are:


Value iteration makes updates using a model of the environment, Q learning works from samples from the environment made by an active agent.


By working from a simulated environment rather than a real one, it may not be clear when an agent is model-free or model-based (or planning rather than acting). However, the way that the simulated environment in the PyTorch example is used is consistent with a model-free method.

Value iteration loops through all possible states and actions for updates independently of any action an agent might take (in fact the agent need not exist). Q learning works with whichever states the agent experienced.


By adding experience replay memory in DQN, Q learning becomes a little bit closer to value iteration, as you can frame the memory as a learned model, plus consider it to be a type of planning (or a "sweep" through states). This is how it is described for instance in DynaQ which is an almost identical algorithm to experience replay as used in DQN when both are used in the simplest versions - see Sutton &amp; Barto chapter 8.

Value iteration value update steps are over an expectation of next states and rewards - it processes the weighted sum $\sum_{r,s'}p(r,s'|s,a)$. Q learning update steps are over sampled next states and rewards - it ends up approximating the same expectation over many separate updates.


Even using large amounts of experience replay memory does not get Q learning the same as value iteration on this issue, samples are not guaranteed perfect. However, in a deterministic environment, this difference is not meaningful. So if you have a deterministic environment, Q learning and value iteration may also be considered a little closer in nature.





* Technically this applies to single-step Q-learning. n-step Q-learning and Q($\lambda$) use different estimates of future expected return, that are related but not the same as the single-step version shown here.