LaTex Text Justification badness penalty
The optimization function is designed in a way to find a solution that:


uses a small number of lines
is balanced (the number of extra spaces at the end of each line should be about the same length)




Lets look first at the linear optimization function, i.e. the sum of all extra spaces.

It is very clear, that the optimal solution under that function is one that uses the minimum number of lines.
However it will not fulfill the second goal.
The number of extra spaces is equal in each solution that uses the same number of lines.

For instance if you want to split the string a.bb.ccc at line-width 6.
The unbalanced solution

a.....
bb.ccc


has the same penalty 5 (number of extra spaces) as the balanced solution

a.bb..
ccc...


. So if you only optimize with a linear function, then you might end up with an unbalanced solution.



Now lets look at the the sum of squares.

The intuition is, that having lines with a lot of extra spaces will get penalized a lot more than lines with less extra spaces.

Using the same example. The unbalanced solution

a.....
bb.ccc


will get a bigger penalty $5^2 + 0^2 = 25$ than

a.bb..
ccc...


which has only a penalty of $2^2 + 3^2 = 13$.

You can actually partially prove that this will give us a balanced solution.

If you fixate the number of lines, than also the number of necessary extra spaces is fixed. Let's assume than we need $n$ lines with $m$ extra spaces. I'll denote $a_i$ for the number of extra spaces in line $i$.

We want to optimize $$\sum_{i = 1}^n a_i^2$$
We know (using the QM-AM inequality), that

$$\sum_{i = 1}^n a_i^2 \ge \frac{(\sum_{i=1}^n a_i)^2}{n} = \frac{m^2}{n} = \text{const}$$

and that we reach equality for $a_i = \frac{m}{n}$ for all $i$.

So the minimum of the optimization function is reached, when every lines has exactly the same number of extra spaces.

Notice, that usually this is not possible (because of the different word lengths in the string). 

But the result supports the argument, that this optimization function ends up with a solution that we human would call "balanced".



The cubic function, and many other ones, will work in exactly the same way. Unbalanced solutions will end up with a higher penalty than balanced solutions.



Btw, $\LaTeX$ uses the sum of squares as penalty. At least Wikipedia claims so.