Self-designed objective for linear regression learning
This question seems really old, it may be helpful to new visitors.

Yes, we can define any objective function and use gradient descent to minimize it. For that the objective function need to have some properties like


It should be convex function else we'll stuck at local minima or may never find the minimum value
It must be differentiable on every point and must have non zero derivative. Else gradient descent will not make progress and get stuck


If cost function have these properties then we can use tensorflow (or other library) to create graph to calculate the cost and then to calculate its gradient using auto-diff algorithm and use gradient descent to minimize it. 

Auto-diff basically breaks calculation of cost function into elementary arithmetic computation like addition, multiplication, division etc and use chain rule to find derivative.