How to evaluate a Deep Q-Network
Q-values represent expected return after taking action $a$ in state $s$, so they do tell you how good it is to take an action in the specific state. Better actions will have larger Q-values. Q-values can be used to compares actions but they are not very meaningful in representing performance of the agent since you have nothing to compare them with. You don't know the actual Q-values so you can't conclude if your agent is approximating well those Q-values or not.

Better performance metric would be the average reward per episode/epoch, or average reward in last $N$ timesteps for continuing tasks. If your agent is improving its performance then it's average reward should be increasing. You said that you have rewards per state and that some of them represent more important criteria then others. You could plot the average reward per episode by doing some kind of weighted linear combination of criteria rewards
\begin{equation}
\bar R = \bar R_1 w_1 + \bar R_2 w_2 + \bar R_3 w_3 + \bar R_4 w_4
\end{equation}
where $\bar R_i$ is the average episode reward for criteria $i$.That way you can provide more importance to some specific criteria in your evaluation.