title: Caret obtain train & cv predictions from model to plot
body: <p>I've trained a simple model:</p>
  
  <pre><code>mySim &lt;- train(Event ~ .,
                 method = 'rf',
                 data = train,
                 tuneGrid = tg)
  </code></pre>
  
  <p>Optimising the two <code>nnet</code> parameters <code>weight_decay</code> and <code>size</code> of the hidden layer. I'm new to trying out <code>caret</code> so what I would usually do is plot the <code>train error</code> and <code>cv error</code> for each model build. To do this, I'd need to have the predictive values of my <code>train</code> and <code>validation</code> pass. </p>
  
  <p>This is the first time I've used cross validation so I'm a little unsure how I can go about getting the predictions from the train and hold-out set at each <code>tuneGrid</code> iteration. </p>
  
  <p>If I have a grid search of length 3 (3 models to build) and 5-fold cross validation I assume I'm going to have 15 sets of train &amp; holdout predictions for each model. </p>
  
  <p>The plot I'm essentially looking to build is: </p>
  
  <p><a href="https://i.stack.imgur.com/hhwyC.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/hhwyC.png" alt="enter image description here"></a></p>
  
  <p>Where my y-axis is a performance metric, lets say entropy loss for the sake of classification with <code>nnet</code> and the <code>size</code> grid search values on the x-axis increases from 0 - max.</p>
  
  <p>Is there a way in which I can extract the predicted values from the train / holdout set during <code>trainControl</code> cross validation? </p>
  
  <p>I've looked through some of the attributes <code>train</code> returns but not sure if I'm missing something. </p>
  
  <p>I know I lack code in this question but hopefully I've explained myself.</p>
  
  <p><strong>Update</strong> 
  I am correct in assuming setting the following parameters in <code>trainControl</code> will return the predictions allowing me to create this plot:</p>
  
  <ul>
  <li><code>returnResamp</code></li>
  <li><code>savePredictions</code> </li>
  </ul>