Methods for calculating signal quality metric
From your question it sounds like you are looking for a SNDR metric.  In that case you are measuring a distortion level by comparing the distorted signal to a reference.
For a discrete-time signal $y$, comparable to reference $r$, the SNDR would be:
$$ SNDR = \frac{E[r^2]}{E[(y-r)^2]} $$