LinearRegression with multiple binary features sometimes performs poorly I have a dataset comprising a number of binary features which are the dummies (as in, pd.get_dummies()) of categorical features. SalePrice is my target variable. 



I'm literally just fitting a sklearn LinearRegression model with that data a thousand times to get an average of the score, and I'm getting a weird result. The relevant bit of my code looks like this:

import numpy as np
scores = np.array([])

for i in range(1000):

    x3_train, x3_test, y3_train, y3_test = train_test_split(
            df3.drop('SalePrice', axis=1),
            df3.SalePrice,
            test_size=0.33
    )

    lr3 = LinearRegression()
    lr3.fit(x3_train, y3_train)

    scores = np.insert(scores, 0, lr3.score(x3_test, y3_test))

print(scores.mean())


Now the weird result is that the average result is super poor, because every so often the model just tanks completely but most of the time performs "reasonably" (still terrible but that's not a surprise as it's incredibly basic and not tuned at all, I'm just comparing the effect of treating a set of features in different ways). For example the first 30 runs generated these scores:

0     5.907010e-01
1     6.044523e-01
2     5.178049e-01
3     5.622240e-01
4     5.810432e-01
5     5.131722e-01
6     5.772946e-01
7     4.674152e-01
8     4.962015e-01
9     4.887872e-01
10    5.144772e-01
11    5.676829e-01
12    5.122566e-01
13    5.453985e-01
14    5.355022e-01
15    5.888459e-01
16    5.552912e-01
17    5.615658e-01
18    5.472429e-01
19    5.810185e-01
20    5.334900e-01
21    5.493619e-01
22    5.567195e-01
23    5.514374e-01
24    4.916478e-01
25    4.580718e-01
26    5.286095e-01
27    5.761865e-01
28    5.638573e-01
29   -1.809208e+24
Name: lr3, dtype: float64


I guess my question is what is likely to be happening on that 30th run through such that the model performs so poorly? I'm comparing this model to others that treat the data differently (e.g. simply encode using .astype('category').cat.codes) and whilst there's relatively minor variations in the "usual" range of scores (they're all sort of 0.44 - 0.63) those other models don't have this occasional complete tanking.
You should always consider normalizing your output to some predefined range, otherwise there is a possibility of the gradients exploding as the loss will be of high magnitudes. It also becomes hard to output such a wide range. Try transforming your output using some StandardScaler, or a RobustScaler if there are significant outliers, and try again.