Relationship between Rewards and Q Value (Graph between Q(s, a) vs episodes) I'm employing the Actor-Critic algorithm. The critic network approximates the action-value function, i.e. $Q(s, a)$, which determines how good a particular state is, when provided with an action.
$Q(s, a)$ is approximated using the backpropagation of the temporal difference error (TD error). We can understand that $Q(s, a)$ has been approximated properly when TD error is minimized, i.e. when it is saturated at lower values.
My question is, when exactly can you say that $Q(s, a)$ is approximated properly, if you don't have TD error, i.e. if you have to plot the graph between $Q(s, a)$ vs episodes, then what would be the optimal behaviour?
Will it be increasing exponential with saturation value around reward values, or increasing exponential with saturation (around any value)?
Follow up: What can be the possible mistake, if the output of Q-value function is around 5x the rewards, and not saturating?
TLDR;
The output of Q-value function will eventually saturate. Can't say when, but it will surely do.
Detailed Answer

if you don't have TD error

It meant, if I don't have logs of the error.

$Q(s, a)$ vs Episodes Graph

To understand, how $Q(s, a)$ behaves as the episodes increase.
I was under a wrong impression that, the $Q(s, a)$ will saturate around the values given by the reward function.
As evident from Loss vs Training Episodes Curve, we can see that loss (TD Error) is almost saturated. 
However, $Q(s, a)$ vs training episodes curve is not saturated yet.

The only explanation for the above two graphs could be given as follows:
The target estimate ($r + \gamma Q(s', a')$) and $Q(s, a)$ are almost similar due to which, the error is quite low. But, $Q(s, a)$ is still nowhere near optimal value $Q^{*}(s, a)$.
Hence, I gave it another shot and made it run for twice training episodes i.e. 20000, and below are the results.
A nearly saturated loss

and a nearly saturating $Q(s, a)$.
.
Note that, the value of $Q(s, a)$ saturating is around 250 - 300 (will run it for more iterations) and it is nowhere around the reward values $ \in [-100, 35]$.
Hence, the $Q(s, a)$ vs episodes will saturate.