title: Scikit correct way to calibrate classifiers with CalibratedClassifierCV
body: <p>Scikit has <a href="http://scikit-learn.org/stable/modules/generated/sklearn.calibration.CalibratedClassifierCV.html" rel="nofollow noreferrer">CalibratedClassifierCV</a>, which allows us to calibrate our models on a particular X, y pair. It also states clearly that <code>data for fitting the classifier and for calibrating it must be disjoint.</code></p>
  
  <p>If they must be disjoint, is it legitimate to train the classifier with the following? </p>
  
  <pre><code>model = CalibratedClassifierCV(my_classifier)
  model.fit(X_train, y_train)
  </code></pre>
  
  <p>I fear that by using the same training set I'm breaking the <code>disjoint data</code> rule. An alternative might be to have a validation set </p>
  
  <pre><code>my_classifier.fit(X_train, y_train)
  model = CalibratedClassifierCV(my_classifier, cv='prefit')
  model.fit(X_valid, y_valid)
  </code></pre>
  
  <p>Which has the disadvantage of leaving less data for training. Also, if <a href="http://scikit-learn.org/stable/modules/generated/sklearn.calibration.CalibratedClassifierCV.html" rel="nofollow noreferrer">CalibratedClassifierCV</a> should only be fit on models fit on a different training set, why would it's default options be <code>cv=3</code>, which will also fit the base estimator? Does the cross validation handle the disjoint rule on its own?</p>
  
  <p><strong>Question: what is the correct way to use CalibratedClassifierCV?</strong></p>