How to solve recurrence $T(n) = 5T(\frac{n}{2}) + n^2\lg^2 n$
You can use the master theorem. 
This theorem allows you to solve some recurrences of the form $T(n) = aT(n/b) + f(n)$.

You need to compare $n^{\log_b a}$ with $f(n)$.
In you case $n^{\log_b a} = n^{\log_2 5}$ and $f(n)=n^2 \log^2 n$.

There are different cases depending on how the above functions compare, but I am only going to discuss the one that is relevant to you (you can find more on Wikipedia).

In your case $f(n) = O(n^{\log_b a - c})$ for some constant $c&gt;0$.
To see this pick, e.g., $c=0.1$ and substitute to obtain:
$n^2 \log^2 n = O(n^{\log_2 5 -0.1})$, which is true since $\log_2 5 - 0.1 &gt; 2$.

The master theorem then tells you that $T(n) = \Theta(n^{\log_b a})$, which in your case is $T(n) = \Theta(n^{\log_2 5})$.