Weird overfitting in linear regression This is really weird, I have a real simple test dataset and built a really simple linear model on it:

smallData = data.head(1000)
# smallData = data

y = smallData['points'].values
X = smallData.drop(['points','country', 'description'], axis=1)

X = pd.get_dummies(X, columns=['designation', 'province', 'region_1', 'region_2', 'variety', 'winery', 'taster_name', 'taster_twitter_handle', 'title'])

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)

regressor = LinearRegression()  
regressor.fit(X_train, y_train)

y_pred = regressor.predict(X_test)


It actually works really fine, as long as I only train it with ~200 rows of data. Once I have 300 rows and more, it returns COMPLETELY wrong predictions for some rows. Like, my scale is 80 to 100, and the prediction says it should be five million. That seems slightly off, especially there is no value of five million in the training data, 100 is the highest.

My data has quite a lot of columns due to the dummy encoding. It seems like it's overfitting?

What can I do to only get reasonable predictions between 80 and 100?
This is called (perfect) collinearity and happens e.g. when the reference category is included in the set of dummy variables. Use pd.get_dummies(..., drop_first=True) to avoid this.