Definition of the Q* function in reinforcement learning
The reward of action $a$  is defined as a stationary probability distribution with mean $q_*(a)$. This is independent of time $t$.
However the estimate of $q_*(a)$ at time $t$, denoted by $Q_t(a)$, is dependent on time $t$

Or are we to understand q∗ as taking the expected reward across all t?

The expectation is not over time, but over a probability distribution with mean $q_*(a)$.
For eg., in the 10-armed bandit problem, the reward for each of the 10 actions comes from a Normal distribution with mean $q_*(a), a= 1,...,10$ and variance 1.