How do I know that the DQN has learnt an appropriate Q function? Is there any sanity check to know whether the Q functions learnt are appropriate in deep Q networks? I know that the Q values for end states should approximate the terminal reward. However, is it normal that Q values for the non-terminal states have higher values than those of the terminal states?

The reason why I want to know whether Q values learnt are appropriate is because I want to apply the doubly robust estimator for off-policy value evaluation. Using doubly robust requires a good Q value estimate to be learnt for each state.
DQN is famous for doing over-approximation on Q function. However, having over approximated Q does not imply that it does not perform well in the environment. (unless it looks ridiculously high) From my experience, high learning rate usually cause over approximated Q, or mistakes made in the code. Best way to check is to see plot of Q function when running on environment to see if it makes any sense. (e.g. It should go lower for bad states and vice versa)