Determining asymptotic notation of a complex function
If you check the definitions, big-O means "bounded from above" (with a fudging constant $c$ multiplying it all, starting at a high enough $n$), you can disregard "constants multiplying" and "slower growing terms"; so you can say:

$\begin{equation*}
 5 n^2 \log n - \dfrac{100 n^2}{\log_4 n^2} + 40
   = O(n^2 \log n)
\end{equation*}$

(could also say $O(n^3)$, or $O(n^4 \log^{10} n)$ for that matter, bound from above).

For a lower bound, the same works fine:

$\begin{equation*}
 5 n^2 \log n - \dfrac{100 n^2}{\log_4 n^2} + 40
   = \Omega(n^2 \log n)
\end{equation*}$

For a lower bound, $\Omega(1)$ or $\Omega(n \log n)$ work fine too.

As you have the same upper/lower bound:

$\begin{equation*}
 5 n^2 \log n - \dfrac{100 n^2}{\log_4 n^2} + 40
   = \Theta(n^2 \log n)
\end{equation*}$

Finding values of $c$ and $N_0$ I leave to you. They can be different for $O$ and $\Omega$!