Summation with floor in upper bound
You want to estimate
$$
S := \sum_{i=1}^{\lfloor \sqrt{n} \rfloor} (n-i).
$$
You can start with the formula
$$
S(m) := \sum_{i=1}^m (n-i) = m \left(n - \frac{m+1}{2}\right).
$$
The formula for $S(m)$ is an increasing function for $m \leq n-1/2$ (as you can check by computing the derivative), and so $\sqrt{n} - 1 &lt; \lfloor \sqrt{n} \rfloor \leq \sqrt{n}$ implies (for $n \geq 2$) that
$$
S(\sqrt{n}-1) &lt; S \leq S(\sqrt{n}+1).
$$
Substituting, we get
$$
n^{3/2} - \frac{3}{2}n + \frac{\sqrt{n}}{2} &lt; S \leq n^{3/2} - \frac{n}{2} - \frac{\sqrt{n}}{2}.
$$
Therefore $S = n^{3/2} - O(n)$. In particular, $S = \Theta(n^{3/2})$.

We can also get an exact formula:
$$
S := \lfloor \sqrt{n} \rfloor \left(n - \frac{\lfloor \sqrt{n} \rfloor + 1}{2}\right).
$$