How does mathematical software evaluate symbolic sums? Wolfram alpha is able to compute this sum:
$$ \sum_{j=1}^n \binom{j}{2} = \frac{1}{6}(n-1)n(n+1). $$

How can Wolfram alpha do it? What kind of algorithm does it use?
It uses algorithms discussed in the book A=B by Petkovšek, Wilf and Zeilberger. These algorithms can compute a closed-form expression for a summation of rational functions (in your case, the rational function is $j(j-1)/2$), or determine that no such closed-form expression exists.