If $n^{\log n}$ is not polynomial or exponential, then what this function is called?
There is no fixed terminology for these types of functions. You might sometimes see "subexponential" or "superpolynomial" used to refer to this kind of behavior.