How do I simplify $O\left({n^2}/{\log{\frac{n(n+1)}{2}}}\right)$
Your expression is 

$$ E = \frac{cn^2}{\log \frac{n(n+1)}{2}}$$

where $c$ is some constant. The simple upper bound for $E$ is

$$ E\le c n^2$$

which implies that $\mathcal{O}(n^2)$. For a better bound 

$$E  = \frac{cn^2}{\log \frac{n(n+1)}{2}} = \frac{cn^2}{ 2 \log n + \log n - \log 2 } $$

Now it is an easy verification that $E$ is $\mathcal{O}(\frac{n^2}{\log n})$. In other words $E$ is $o(n^2)$(weaker statement as compare to the previous one).