Apart from the state and state-action value functions, what are other examples of value functions used in RL?
Advantage function: $A(s,a) = Q(s,a) - V(s)$
More interesting is the General Value Function (GVF), the expected sum of the (discounted) future values of some arbitrary signal, not necessarily reward. It is therefore a generalization of value function $V(s)$. The GVF is defined on page 459 of the 2nd edition of Sutton and Barto's RL book as
$$v_{\pi,\gamma,C}(s) =\mathbb{E}\left[\left.\sum_{k=t}^\infty\left(\prod_{i=t+1}^k \gamma(S_i)\right)C_{k+1}\right\rvert S_t=s, A_{t:\infty}\sim\pi\right]$$
where $C_t \in \mathbb{R}$ is the signal being summed over time.
$\gamma(S_t)$ is a function $\gamma: \cal{S}\to[0,1]$ allowing the discount rate to depend upon the state. Sutton and Barto call it the termination function. Some call it the continuation function.
Also of note are the differential value functions. These are used in the continuing, undiscounted setting. Because there is no discounting, the expected sum of future rewards is unbounded. Instead, we optimize the expected differential reward $R_{t+1}-r(\pi)$, where $r(\pi)$ is the average reward under policy $\pi$.
$$v_{\pi,\,diff}(s) = \sum_a \pi(a|s) \sum_{s',r} p(s',r|s,a)\left[r-r(\pi)+ v_{\pi,\,diff}(s')\right]$$
$$v_{*,\,diff}(s) = \max_a \sum_{s',r} p(s',r|s,a)\left[r-\max_\pi r(\pi)+ v_{*,\,diff}(s')\right]$$
The differential value functions assume that a single fixed value of $r(\pi)$ exists. That is, they assume the MDP is &quot;ergodic.&quot; See section 10.3 of Sutton and Barto for details.