Understanding advantage functions
1. Why are the q-values of different actions very close to each other for a given state ? 

I'm going to explain this with a small example.

Consider the game of "Catch". Fruits(circular) keep falling from the top of the screen (vertically) and the agent(square) needs to just align itself to the fruit to get the reward. There are three actions that it can take : move left , stay , move right. Let's say that a2 refers to not moving the paddle, a3 refers to moving right and a1 refers to moving left. The image on the screen is to be used as the state in the MDP formalism.



Let's say we take a sub-optimal action a3(move right) and move to the next state. Then the best action in that state would be to move left (a1) and then execute the optimal action. So, the only cost difference between the actions a2 and a3 will be the two steps wasted to go and come back.

If there is no negative reward for taking the sub-optimal action, then the agent has no incentive to choose the optimal action. So, the negative rewards for taking the sub-optimal action should be high enough that the agent is discouraged from doing it. I've tried to put this intuition mathematically here. This could explain why q-values are so close to each other.

Then, the optimal Q* function satisfies the following :



2. Is there a bound on this gap ?
The bound that I've shown is approximate. The technical term for this difference is "Action Gap". A much better analysis is available here : Increasing the Action Gap: New Operators for Reinforcement Learning

3. Why are the q-values of different states very close to each other for a given action ?

This should be fairly obvious. Consider two different cases. In the first one, the fruit falls from the top left corner and in the second one, the fruit falls from the top right corner. The agent in both cases is at the center. Then, consider the action of moving left. This action will give a high q-value in the first case and a low q-value in the second case.

4. How to solve the action gap problem ?

Let's define the average reward in a state to be $V$. $V$ is approximately the average of Q-values of all the actions in a given state. If we now subtract this $V$ from each of the $Q$ values, we get a quantity which describes how good the action is compared to the average of all the action. This is called as the advantage of the action. In some sense, this is approximately the action gap. So, instead of predicting the q-values, we are directly predicting the action gap itself.

Link : A3C paper