title: Weird linear regression learning curve
body: <p>I'm trying to build a prediction model for apartments price. I use python scikit-learn toolset. I'm using a dataset having total floor area and location of the apartment, which I have converted to dummy features. So the dataset looks like this:
  <a href="https://i.stack.imgur.com/OmGL2.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/OmGL2.png" alt="enter image description here"></a>
  Then I build a learning curve to see how the model is doing. 
  I build the learning curve this way:</p>
  
  <pre class="lang-py prettyprint-override"><code>from matplotlib import pyplot as plt
  from sklearn.linear_model import LinearRegression
  from sklearn.model_selection import learning_curve
  
  model = LinearRegression()
  training_sizes, training_scores, validation_scores = learning_curve(
      estimator = model,
      X = X_train,
      y = y_train,
      train_sizes = np.linspace(5, len(X_train) * 0.8, dtype = int), 
      cv = 5
  )
  line1, line2 = plt.plot(
      training_sizes, training_scores.mean(axis = 1), 'g', 
      training_sizes, validation_scores.mean(axis = 1), 'r')
  plt.legend((line1, line2), ('Training', 'Cross-validation'))
  </code></pre>
  
  <p>The picture I see is somewhat confusing:
  <a href="https://i.stack.imgur.com/Ixbpx.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Ixbpx.png" alt="Learning curve on training and cross-validation sets"></a>
  Anomalies I see here are:</p>
  
  <ol>
  <li>Huge error on cross-validation set</li>
  <li>Error not steadily decreasing on training examples number growth. </li>
  </ol>
  
  <p>Is it normal? </p>
  
  <p>Learning curve of training set only is also not so smooth but at least the error isn't that huge:
  <a href="https://i.stack.imgur.com/0kiFr.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/0kiFr.png" alt="Learning curve on training set"></a></p>
  
  <p>Also I tried to add to add polynomial features of 2nd degree. But this didn't make the model perform any different. And because I have a lot of categorical features (total 106) it takes quite long even for 2nd degree polynomial. So I didn't try for higher degrees. </p>
  
  <p>Also I tried to build a model using as simple cost function and gradient descent as possible using Octave. The result with weird error was same.  </p>
  
  <p>Update:
  Thanks to <a href="https://stackoverflow.com/a/60734720/200160">tolik</a> I made several amendments:</p>
  
  <p><strong>Data preparation:</strong>
  Categorical data are independent. So I can't combine them into one feature.
  Features were scaled using StandardScaler(). Thank you for that.</p>
  
  <p><strong>Feature extraction:</strong>
  After features transformation with PCA I found out one new feature has explained variance ratio over 99%. Though it's strange I used only this one. That also allowed to increase polynomial degree though it didn't increase performance.</p>
  
  <p><strong>Model selection:</strong>
  I tried several different models but none seem to perform better than LinearRegression. Interesting thing - all models perform worse on full data set. Probably it's because I sorted by price and higher prices are rather outliers. So when I start training sets on 1000 samples and go to the maximum, I get this picture (for nearly all models):
  <a href="https://i.stack.imgur.com/wsS30.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/wsS30.png" alt="enter image description here"></a></p>