How to reduce the error in linear regression
Answer based on the comment: 


  Intention here is to reduce the error by all means and considerations



Choice of model: First let's address the obvious assumption: linear regression is a model which requires the response variable to be expressed as a linear combination of the independent variables. In order to improve performance in general one must make sure that these constraints are satisfied. If not it's often worth looking into other models or finding a way to "linearize" the data.
Data cleaning: depending on the size of the data, linear regression can be very sensitive to outliers. If it makes sense for the problem, outliers can be discarded in order to improve the quality of the model. Of course one shouldn't remove points just because it decreases the error, this has to be done sensibly with respect to the characteristics of the task.
Feature engineering: it's worth analyzing/experimenting which of the independent variables actually help obtaining a good model. For instance redundant variables may decrease performance. It's also possible that variables can be expressed differently or transformed in a way which improves performance.