Why is finding the closest pair of points by brute force search O(n^2)?
It's not clear what you think $O(n^2)$ means.  $O(n^2)$ means that there is a constant $c$ so that, when $n$ is large enough, your algorithm takes no more than $cn^2$ time.

As you observed, the algorithm takes  $(n-1)+(n-2) + \ldots + 1 $ time.  This is equal to $\frac12 (n^2-n)$.  This quantity is less than $2n^2$ whenever $n$ is larger than $10$.  So it is indeed $O(n^2)$.

Another way to understand $O(n^2)$ is: if $n$ doubles, the algorithm will take about four times as long.   For $n=10$, your algorithm takes $9+8+\ldots + 1 = 45$ steps.  When $n=20$, it takes $ 19+18+\ldots + 1 = 190$ steps.  That is indeed pretty close to $4\cdot 45=180$ steps.  Similarly when $n=40$ it takes $780$ steps, which is again pretty close to $4\cdot 190 = 760$.