Upper bound of algorithm with flooring
If a number is of the form $n = 2^{2^k}$ then you never encounter the floor as you open up the recursion, and then there is no problem. If you're interested in a general upper bound, you can prove by induction that $T(n)$ is monotone in $n$: if $n \leq m$ then $T(n) \leq T(m)$. Now given $n$, find some $m \leq n^2$ which is of the form $2^{2^k}$, and then
$$ T(n) \leq T(m) = O(\log m \log\log m) = O(\log n \log\log n). $$