Explanation behind actor-critic algorithm in pytorch example?
Pytorch provides a good example of using actor-critic to play Cartpole in the OpenAI gym environment. 

I'm confused about several of their equations in the code snippet found at https://github.com/pytorch/examples/blob/master/reinforcement_learning/actor_critic.py#L67-L79: 

saved_actions = model.saved_actions
value_loss = 0
rewards = []
for r in model.rewards[::-1]:
    R = r + args.gamma * R
    rewards.insert(0, R)
rewards = torch.Tensor(rewards)
rewards = (rewards - rewards.mean()) / (rewards.std() + np.finfo(np.float32).eps)
for (action, value), r in zip(saved_actions, rewards):
    action.reinforce(r - value.data.squeeze())
    value_loss += F.smooth_l1_loss(value, Variable(torch.Tensor([r])))
optimizer.zero_grad()
final_nodes = [value_loss] + list(map(lambda p: p.action, saved_actions))
gradients = [torch.ones(1)] + [None] * len(saved_actions)
autograd.backward(final_nodes, gradients)
optimizer.step()


What do r and value mean in this case? Why do they run REINFORCE on the action space with the reward equal to r - value? And why do they try to set the value so that it matches r? 

Thanks for your help!


First the rewards a collected for a time, along with the state:action that resulted in the reward
Then r - value is the difference between the expected reward and actual
That difference is used to adjust the expected value of that action from that state


So if in state "middle", the expected reward for action "jump" was 10  and the actual reward was only 2, then the AI was off by -8 (2-10).  Reinforce means "adjust expectations".  So if we adjust them by half, we'll new expected reward is 10-(8 *.5), or 6.  meaning the AI really thought it would get 10 for that, but now it's less confident and thinks 6 is a better guess.   So if the AI is not off by much, 10 - ( 2 *.5) = 9, it will adjust by a smaller amount.  
