What, exactly, does the REINFORCE update equation mean? I understand that this is the update for the parameters of a policy in REINFORCE:
$$
\Delta \theta_{t}=\alpha \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right) v_{t}
$$
Where  is usually the discounted future reward and  (|) is the probability of taken the action that the agent took at time . (Tell me if something is wrong here)
But I don’t understand how that is passed into a neural network for back propagation.
I have this pseudocode
probs = policy.feedforward(state)
This returns the probabilities if taking each actions, like: [0.6,0.4]
action = choose_action_from(probs) this will return the index of the probability chosen. For example, if it chose 0.6, the action would be 0.
Then later when it is time to update, is it:
gradient = policy.backpropagate(total_discounted_reward*log(probs[action])
policy.weights += gradient

Is this the right way to calculate the derivative of the loss and backpropagate it? And I only backpropagate this through one output neuron?
If you need more explanation, I have this question on SO.
The loss function you are looking for is cross entropy loss. The 'label' that you use is the action you took at the time point you are updating for.