Training a neural net without labels in Reinforcement Learning
What you are looking for is DQN: Deep Q-Network.

You want to have a look at the example code, but the idea is as follows:


Input: The observation
Output: Reward - the action can either be in the input or better one reward-prediction per possible action, if the action is discrete. 


For final states, you know the true reward. Hence your target is clear (see line 112). But if this is not the case, you need to adjust the target to the observed reward and the discounted expected reward (line 113ff).

Extremely important: The output layers activation function. Don't use softmax, as it is unlikely that your rewards sum up to 1. Don't use ReLU if the reward can be negtive. Simply using a linear output makes most sense in most cases.