Is Expected SARSA an off-policy or on-policy algorithm?
Expected SARSA can be used either on-policy or off-policy.

The policy that you use in the update step determines which it is. If the update step uses a different weighting for action choices than the policy that actually took the action, then you are using Expected SARSA in an off-policy way.

Q-learning is a special case of Expected SARSA, where the target policy is greedy with respect to the action values, so there is only ever one $r_{t+1} + \gamma \text{max}_{a'} Q(s_{t+1}, a')$ term to add with a probability $1$.

You can also use Expected SARSA, similarly to SARSA, where the behaviour policy and target policy are identical. It is not identical to SARSA though, because it calculates the TD Target over all possible actions $r_{t+1} + \gamma \sum_{a'} \pi(a'|s_{t+1}) Q(s_{t+1}, a')$

You can construct Expected SARSA updates where $\pi(a|s)$ is different when selecting which action to explore in the environment (behaviour) and when updating the Q values (target). For instance, you can decide to explore using $\epsilon$-greedy with $\epsilon=0.1$ and update the value function with $\epsilon=0.01$.