Could anyone prove why O(n^(log n)) < O((log n)^n)?
Using $ &lt; $ with the big-oh notation is a bit of abuse of notation.
Formally, I take your statement to mean $\forall f(n) \in O(n^{\log n}), \; f(n) \in O((\log n)^n)$. 

Let $f(n) \in O(n^{\log n})$, by definition of big-oh we know that the function is eventually bounded from above by $c n^{\log n}$ for some constant $c &gt; 0$.
Then, for sufficiently large $n$,
$$
f(n) \le c n^{\log n} = c 2^{\log(n^{\log n})} = c2^{\log^2 n} &lt; c2^n &lt; c (\log n)^n,
$$
showing that $f(n) \in O((\log n)^n)$.

In particular the constant c in the last term can be made arbitrarily small. Indeed $n^{\log n} \in o((\log n)^n)$. A simple way to see this is by taking the limit of the ratio of the two functions:

$$
\lim_{n \to \infty} \frac{n^{\log n}}{(\log n)^n} =
\lim_{n \to \infty} \frac{2^{\log^2 n}}{2^{n \log n}} =
2^{\lim_{n \to \infty} \log^2 n - n \log n} = 0
$$