Relations of some exponential functions -- which is correct?
(1) is not the correct answer.  Perhaps there is an error in the book, or an error in what you entered.  As you correctly concluded, $f(n) = 3 e^{n^{1/2} \log n}$ and $h(n) \approx e^{n \log n}$ (by Stirling's approximation), so it is not true that $h(n) = O(f(n))$.

I can't tell whether you've drawn the correct conclusions about the pairwise relationships between these three functions.  You wrote that $f(n) &lt; g(n) &lt; h(n)$, but this is not correct.  For instance, $f(1) = 3$, $g(1) = 3$, $h(1) = 1$.  Those inequalities don't hold for all $n$ -- but if we care about big-O notation, then that's not the right question anyway, since big-O notation relates to the asymptotic behavior of these functions (as $n\to \infty$).

The right question is whether $f(n) = O(g(n))$, whether $g(n) = O(f(n))$ (be careful here!  better double-check your work on this one another time), and so on.

See also Sorting functions by asymptotic growth and How does one know which notation of time complexity analysis to use?.  They should answer your remaining questions.