What is a time-step in a Markov Decision Process?
In a Markov Decision Process (MDP) model, we define a set of states ($S$), a set of actions ($A$), the rewards ($R$), and the transition probabilities $P(s' \mid s, a)$. The goal is to figure out the best action to take in each of the states, i.e. the policy $\pi$.
Policy
To calculate the policy we make use of the Bellman equation:
$$V_{i+1}(s)=R(s)+\gamma \max _{a \in A}\left(\sum_{s^{\prime} \in S} P\left(s^{\prime} \mid s, a\right) V_{i}\left(s^{\prime}\right)\right)$$
When starting to calculate the values we can simply start with:
$$V_{1}(s)=R(s)$$
To improve this value, we should take into account the next action, which can be taken by the system and will result in a new reward:
$$V_{2}(s)=R(s)+\gamma \max _{a \in A}\left(\sum_{s^{\prime} \in S} P\left(s^{\prime} \mid s, a\right) V_{1}\left(s^{\prime}\right)\right)$$
Here you take into account the reward of the current state $s$: $R(s)$, and the weighted sum of possible future rewards. We use $P(s' \mid s, a)$ to give the probability of reaching state $s'$ from $s$ with action $a$. $\gamma$ is a value between $0$ and $1$ and is called the discount factor because it reduces the importance of future rewards since these are uncertain. An often-used value is $\gamma = 0.95$.
When using value iteration this process is continued until the value function has converged, which means that the value function does not change significantly when doing new iterations:
$$\left\|V_{i+1}(s)-V_{i}(s)\right\|&lt;\epsilon, \; \forall_{s \in S},$$
where $\epsilon$ is a really small value.
Discounted sum of future rewards
If you look at the Bellman equation and execute it iteratively you'll see:
$$ {\scriptstyle  V(s)=R(s) + \gamma \max _{a \in A}\left(\sum_{a \in A} P\left(s^{\prime} \mid s, a\right)\left[R\left(s^{\prime}\right) +  \gamma \max _{a \in A}\left(\sum_{s^{\prime \prime} \in S} P\left(s^{\prime \prime} \mid s^{\prime}, a\right)\left(R\left(s^{\prime \prime}\right) + \gamma \max _{a \in A}\left(\sum_{s^{\prime \prime \prime} \in S} P\left(s^{\prime \prime \prime} \mid s^{\prime \prime}, a\right) V\left(s^{\prime \prime \prime}\right)\right)\right]\right)\right.\right. }$$
This is like (without transition functions):
$$R(s)+\gamma R\left(s^{\prime}\right)+\gamma^{2} R\left(s^{\prime \prime}\right)+\gamma^{3} R\left(s^{\prime \prime \prime}\right)+\ldots$$
To conclude
So when we start in state s we want to take the action that gives us the best total reward taking into account not only the current, or next state, but all possible next states until we reach the goal. These are the time steps you refer to, i.e. each action taken is done in a time step. And when we learn the policy we try to take into account as many time steps as possible to choose the best action.

You can find quite a large number of examples if you search on the internet, for example, in the slides of the CMU, the UC Berkeley or the UW.