What algorithms are considered reinforcement learning algorithms?
The dynamic programming algorithms (like policy iteration and value iteration) are often presented in the context of reinforcement learning (in particular, in the book Reinforcement Learning: An Introduction by Barto and Sutton) because they are very related to reinforcement learning algorithms, like $Q$-learning. They are all based on the assumption that the environment can be modelled as an MDP.
However, dynamic programming algorithms require that the transition model and reward functions of the underlying MDP are known. Hence, they are often referred to as planning algorithms, because they can be used to find a policy (which can be thought of as plan) given the &quot;dynamics&quot; of the environment (which is represented by the MDP). They just exploit the given &quot;physical rules&quot; of the environment, in order to find a policy. This &quot;exploitation&quot; is referred to as a planning algorithm.
On the other hand, $Q$-learning and similar algorithms do not require that the MDP is known. They attempt to find a policy (or value function) by interacting with the environment. They eventually infer the &quot;dynamics&quot; of the underlying MDP from experience (that is, the interaction with the environment).
If the MDP is not given, the problem is often referred to as the full reinforcement learning problem. So, algorithms like $Q$-learning or SARSA are often considered reinforcement learning algorithms. The dynamic programming algorithms (like policy iteration) do not solve the &quot;full RL problem&quot;, hence they are not always considered RL algorithms, but just planning algorithms.
There are several categories of RL algorithms. There are temporal-difference, Monte-Carlo, actor-critic, model-free, model-based, on-policy, off-policy, prediction, control, policy-based or value-based algorithms. These categories can overlap. For example, $Q$-learning is a temporal-difference (TD), model-free, off-policy, control and value-based algorithm: it is based on an temporal-difference (TD) update rule, it doesn't use a model of the environment (model-free), it uses a behavioural policy that is different than the policy it learns (off-policy), it is used to find a policy (control) and it attempts to approximate a value function rather than directly the policy (value-based).