How does mathematical software evaluate symbolic sums?
It uses algorithms discussed in the book A=B by Petkovšek, Wilf and Zeilberger. These algorithms can compute a closed-form expression for a summation of rational functions (in your case, the rational function is $j(j-1)/2$), or determine that no such closed-form expression exists.