Minimize absolute values of errors instead of squares
Both the square and the absolute value should work okay for gradient descent but the square will work better.  For calculus based methods, the absolute value method may be intractable.

If you are using gradient descent methods then the square works very nicely in that it tends to form a U shape which takes big steps away from the minimum and smaller steps close to the minimum. Conversely, absolute values tend to form V shapes which take roughly the same step size. The result is that squares have better convergence properties.

Now, calculus based methods tend to take the derivative of a function and set it equal to zero.  Notice that U shapes tend to have well defined derivatives equal to zero at the minima, which works perfectly. By contrast, the V shape has an undefined derivative at its minima which tend to make conjugate gradient methods un-invertible.

Hope this helps!