Is $n^2 \log n$ in $O (n^2)$ am confused
$n^2\log(n)$ isn't $O(n^2)$. You need to have a constant $c$, so since $\log(n)$ tends to infinity, for any fixed $c$ you will choose, at some point $\log(n)$ will pass $c$, that is: $\forall n&gt;n_0: c &lt; \log(n)$ for some $n_0$. This directly means that $\forall n&gt;n_0: cn^2  &lt; n^2 \log(n)$ which means that $n^2 \log(n)$ is asymptotically bigger than $cn^2$.