title: Why is cross_val_predict not appropriate for measuring the generalisation error?
body: <p>When I train a SVC with cross validation,</p>
  
  <pre><code>y_pred = cross_val_predict(svc, X, y, cv=5, method='predict')
  </code></pre>
  
  <p><code>cross_val_predict</code> returns one class prediction for each element in X, so that <code>y_pred.shape = (1000,)</code> when <code>m=1000</code>.
  This makes sense, since <code>cv=5</code> and therefore the SVC was trained and validated 5 times on different parts of X. In each of the five validations, predictions were made for one fifth of the instances (<code>m/5 = 200</code>). Subsequently the 5 vectors, containing 200 predictions each, were merged to <code>y_pred</code>.  </p>
  
  <p>With all of this in mind it would be reasonable for me to calculate the overall accuracy of the SVC using <code>y_pred</code> and y.</p>
  
  <pre><code>score = accuracy_score(y, y_pred)
  </code></pre>
  
  <p>But (!) the documentation of <code>cross_val_predict</code> states:</p>
  
  <blockquote>
    <p>The result of cross_val_predict may be different from those obtained
    using cross_val_score as the elements are grouped in different ways.
    The function cross_val_score takes an average over cross-validation
    folds, whereas cross_val_predict simply returns the labels (or
    probabilities) from several distinct models undistinguished. Thus,
    cross_val_predict is not an appropriate measure of generalisation
    error.</p>
  </blockquote>
  
  <p>Could someone please explain in other words, why <code>cross_val_predict</code> is not appropriate for measuring the generalisation error e.g. via <code>accuracy_score(y, y_pred)</code>?</p>
  
  <hr>
  
  <p><strong>Edit:</strong></p>
  
  <p>I first assumed that with <code>cv=5</code> in each of the 5 validations predicitons would be made for all instances of X. But this is wrong, predictions are only made for 1/5 of the instances of X per validation. </p>