learning curve Sklearn
[After Update1/2]

A learning curve shows how error changes as the training set size increases. One basically change the size of training data points and measure a desired score and compare it against a fixed test set to see how it generalizes. For you the utmost important part to note is the fixed test set. If you were to change the test set, that you could theoretically, how could you evaluate the performance of the same model by changing the size of the training size (because you are changing both datasets at the same time)?!

That is why you pass the the whole X,y in The Sklearn's learning curve, and it returns the tuple containing train_sizes, train_scores, test_scores. As far as I understood, it does a nested validation cross-validation. It basically takes the whole X,y, and split into train/test  (keeping the test data strictly independent and fix in size), depending on how you pass the cv parameter, and keep increasing the training size and record the performance scores for plotting the learning curve. This ensures a proper measure of the optimal model's performance. 

Concerning your Update 1: Having just discussed the details, then your Update 1 example scenario would be wrong. We train a model (no matter how), and would like to evaluate its performance on a hold-out dataset. If you change your hold-out dataset, you wouldn't know whether changing the training size led to the change in test score or the variability of the hold-out dataset itself! 

Concerning your Update 2: This is exactly what we have been discussing so far. No matter what train_size is, the test_size would remain 1914, as long as the same cv method is used.

Example: To further clarify let's imagine we have the following scenario:

train_sizes = [1, 50, 100, 200, 400, 600]
# X.shape: (1000, 5)
# len(train_sizes)=6


If we were to use the following settings in learning_curve class in sklearn:

cv = ShuffleSplit(n_splits=10, test_size=0.2, random_state=0)
learning_curve(estimator, X, y, cv=cv, n_jobs=n_jobs, train_sizes=train_sizes)


The learning_curve returns the train_sizes, train_scores, test_scores  for six points as we have 6 train_sizes. And for these points the train_sizes and test_size would look like this:


Point 1: train_size: 1,   test_size:200
Point 2: train_size: 50,  test_size:200
Point 3: train_size: 100, test_size:200
Point 4: train_size: 200, test_size:200
Point 5: train_size: 400, test_size:200
Point 6: train_size: 600, test_size:200


I think train_sizes are clear (taken from the list we provided). The test_size remains fixed, and it is decided because we chose test_size=0.2 in the cv (recall that we have had 1000 data points, so 20% of it would be -> 1000 * 0.2 = 200), and it has nothing to do with our train_sizes!! This is the hold-out set that we keep fixed during this experiment. Hope this last example clarifies it altogether. ;-)

I would still keep that blogpost here as reference for others.