I'm doing a personal project on regression and I hope I can get some advice from you on several problems. The dataset I'm having is about cuisine, shape is 180x9, and the 2 continuous variables are "prep_time" and "cook_time". I want to predict cook_time based on other variables. I one hot encoded other categorical variables and get a total of ~30 features in the end (and only prep_time is continuous). This is the dataset before processing:

enter image description here

The baseline model with linear regression, SVR and Random forest doesn't perform really well (The absolute r2 scores are <0.1 for all 3 models). But when I added a feature total_time = prep_time+cook_time, and used MinMaxscale on prep_time and total_time, the RF model improves to r2 of ~0.3. However, LinReg results in ~0 mse and 1.0 r2 on the test set, surprisingly.

Thus, I tried to analyze this LR model more. I used sklearn.feature_selection.SelectFromModel function to get the important features from this Linear Regression model, and it shows that only the prep_time and total_time are important for the model. In fact, the model trained using only these 2 variables perform just as well. VIF of these 2 variables are ~7.0, which implies multicollinearity. However, if I remove either of these 2 variables, and train on only the other feature, the model doesn't perform that well.

My question is, is this still a good model? What could explain/potentially be a problem with this LinReg model when it has such a low mse and high r2? Should I use another algorithm instead?

Thank you!

  • $\begingroup$ Something is weird if you get $R^2=0$ in the out-of-sample data. $\text{//}$ $R^2$ loses its "proportion of variance explained" in nonlinear models like random forest and support vector regression, so it is just a reformulation of mean squared error that does not give the desired answer to the "Is this good enough?" question. $\endgroup$
    – Dave
    Jul 21 at 15:07
  • $\begingroup$ @Dave so in this case what are the metrics we can use? Is it just mse? $\endgroup$ Jul 21 at 15:25
  • 1
    $\begingroup$ $MSE$ and $MAE$ are completely viable metrics for this. $R^2$ is not a wrong metric, but keep in mind that it is just a function of the $MSE$. Any model that performs better on $MSE$ (on the same data!) will perform better on $R^2$, and any model that performs better on $R^2$ will perform better on $MSE$. The danger I see with $R^2$ is that it gets us thinking like grades in school. A useful model might have $R^2 = 0.2$, but that runs the risk of getting us to think that we got a $20\%$ and an F grade. Likewise, $R^2 > 0.9$ might looks like an A grade, but such performance might be pedestrian. $\endgroup$
    – Dave
    Jul 21 at 15:28

I suspect the issue is:

But when I added a feature total_time = prep_time + cook_time

Your response variable is cook_time. So it is nonsense to include it as a predictor. Suppose your model is something like:

$$y = x_1 + x_2 + x_3 + x_4$$

where $y$ is cook time, and $x_1$ is prep time.

So let us introduce another variable to represent total time: $z = y + x_1$, or $x_1 = z - y$. Now we substitute this into the model and we get:

$$y = z - y + x_2 + x_3 + x_4$$

So now we have $y$ on both sides of the model equation. This is going to lead to a perfect fit, which means that $R^2$ will be 1.0) and there will be no residuals (so that $MSE$ will be zero).

Moreover, let's just say that you did proceed with such a model. Now, when it comes to making predictions on new (out of sample) data, for which you obviously don't know what the cook time is, how can you form the total_time feature to predict on the new data ?

  • $\begingroup$ This makes sense, thank you so much! $\endgroup$ Jul 21 at 15:26
  • $\begingroup$ You're very welcome. Glad it could help :) $\endgroup$ Jul 21 at 15:28

Your Answer

