Does Master Theorem apply to $T(n) = 4T(n/2) + n^2 \log n$
You can expand the recurrence $T(n) = 4T(n/2) + n^2 \log n$ directly:
\begin{align}
T(n) &amp;= n^2 \log n + 4T(n/2) \\ &amp;=
n^2 \log n + 4(n/2)^2 \log(n/2) + 16T(n/4) \\ &amp;=
n^2 \log n + 4(n/2)^2 \log(n/2) + 16(n/4)^2 \log(n/4) + 64T(n/8)
\end{align}
and so on. We can simplify the terms: they are $n^2 \log n$, $n^2 \log(n/2) = n^2 (\log n - 1)$, $n^2\log (n/4) = n^2(\log n - 2)$, and so on. Unrolling the recurrence all the way gives
\begin{align}
T(n) &amp;= n^2 (\log n + \log (n/2) + \cdots + \log(n/(n/2)) + 4^{\log n}T(1) \\ &amp;=
n^2 \sum_{k=1}^{\log n} k + n^2 T(1) \\ &amp;=
\frac{1}{2} n^2 \log n(\log n - 1) + n^2 T(1) \\ &amp;= \Theta(n^2\log^2 n).
\end{align}
This is also what the master theorem gives (case 2 in the Wikipedia version).