Is log(n) equivalent to (log(n))^x for big-O analysis?
$f(n)$ is not $O(n^2\log n)$

If we denote $f(n) = (n\log n+1)^2+(log n+1)(n^2+1)$
then:
$$f(n) = n^2 \log^2n + 2n\log n + 1 + n^2\log n + n^2 + \log n + 1 \geq n^2log^2n$$

since all the other terms are obviously greater than 0. Therefore by definition $f(n) = \Omega (n^2log^2n)$. Which strictly means it cannot be bounded from above by anything asymptotically greater.

Note that it also holds that $f(n) \leq 10n^2log^2n$, so $f(n) = O(n^2log^2n)$ and also $\Theta$ of the same.