How do updates in SARSA and Q-learning differ in code? The update rules for Q-learning and SARSA each are as follows:

Q Learning:

$$Q(s_t,a_t)←Q(s_t,a_t)+α[r_{t+1}+γ\max_{a'}Q(s_{t+1},a')−Q(s_t,a_t)]$$

SARSA:

$$Q(s_t,a_t)←Q(s_t,a_t)+α[r_{t+1}+γQ(s_{t+1},a_{t+1})−Q(s_t,a_t)]$$

I understand the theory that SARSA performs 'on-policy' updates, and Q-learning performs 'off-policy' updates.

At the moment I perform Q-learning by calculating the target thusly: 

target = reward + self.y * np.max(self.action_model.predict(state_prime))


Here you can see I pick the maximum for the Q-function for state prime (i.e. greedy selection as defined by maxQ in the update rule). If I were to do a SARSA update and use the same on-policy as used when selecting an action, e.g. ϵ-greedy, would I basically change to this:

if np.random.random() &lt; self.eps:
    target = reward + self.y * self.action_model.predict(state_prime)[random.randint(0,9)]
else:
    target = reward + self.y * np.max(self.action_model.predict(state_prime))


So sometimes it will pick a random future reward based on my epsilon greedy policy?
Picking actions and making updates should be treated as separate things. For Q-learning you also need to explore by using some exploration strategy (e.g. $\epsilon$-greedy).

Steps for Q-learning:
1) initialize state $S$
For every step of the episode:
2) choose action $A$ by some exploratory policy (e.g. $\epsilon$-greedy) from state $S$
3) take action $A$ and observe $R$ and $S'$
4) do the update $Q(S, A) = Q(S, A) + \alpha(R + \gamma*\max_aQ(S', a) - Q(S, A))$
5) update the state $S = S'$ and keep looping from step 2 until the end of episode

Steps for Sarsa:
1) initialize state $S$
2) initialize first action $A$ from state $S$ by some exploratory policy (e.g. $\epsilon$-greedy)
For every step of the episode:
3) take action $A$ and observe $R$ and $S'$
4) choose action $A'$ from state $S'$ by some exploratory policy (e.g. $\epsilon$-greedy)
5) do the update $Q(S, A) = Q(S, A) + \alpha(R + \gamma * Q(S', A') - Q(S, A))$
6) update state and action $S = S'$, $A = A'$ and keep looping from step 3 until end of the episode