Intuitively, how does it make sense to take an action $A'$ when the environment already ended?
Intuitively, how does it make sense to take an action A' when the environment already ended?


It doesn't make sense, in that nothing can happen once the agent reaches a terminal state. However, it is often modelled as an &quot;absorbing state&quot; where the action is unimportant (either null or value ignored) with value by definition of $0$.

And you can't set all the terminal values to zero because you are using function approximators

The value is zero by definition. There is no need to approximate it. So don't use function approximators for action values in terminal states.  When $S'$ is terminal, the update becomes:
$Q(S,A) \leftarrow Q(S,A) + \alpha(R - Q(S,A))$
Look at any implementation of Q learning and you will see a conditional calculation for the update value, that uses some variant of the above logic when $S'$ is terminal. For OpenAI Gym environments for instance, it will use the done flag.