title: Caret package Custom metric
body: <p>I'm using the caret function "train()" in one of my project and I'd like to add
  a "custom metric" F1-score. I looked at this url <a href="http://cran.r-project.org/web/packages/caret/caret.pdf" rel="noreferrer">caret package</a>
  But I cannot understand how I can build this score with the parameter available.</p>
  
  <p>There is an example of custom metric which is the following:</p>
  
  <pre><code>## Example with a custom metric
  madSummary &lt;- function (data,
  lev = NULL,
  model = NULL) {
  out &lt;- mad(data$obs - data$pred,
  na.rm = TRUE)
  names(out) &lt;- "MAD"
  out
  }
  robustControl &lt;- trainControl(summaryFunction = madSummary)
  marsGrid &lt;- expand.grid(degree = 1, nprune = (1:10) * 2)
  earthFit &lt;- train(medv ~ .,
  data = BostonHousing,
  method = "earth",
  tuneGrid = marsGrid,
  metric = "MAD",
  maximize = FALSE,
  trControl = robustControl)
  </code></pre>
  
  <hr>
  
  <p>Update:</p>
  
  <p>I tried your code but the problem is that it doesn't work with multiple classes like with the code below (The F1 score is displayed, but it is weird) I'm not sure but I think the function F1_score works only on binary classes</p>
  
  <pre><code>library(caret)
  library(MLmetrics)
  
  set.seed(346)
  dat &lt;- iris
  
  ## See http://topepo.github.io/caret/training.html#metrics
  f1 &lt;- function(data, lev = NULL, model = NULL) {
  
  print(data)
    f1_val &lt;- F1_Score(y_pred = data$pred, y_true = data$obs)
    c(F1 = f1_val)
  }
  
  # Split the Data into .75 input
  in_train &lt;- createDataPartition(dat$Species, p = .70, list = FALSE)
  
  trainClass &lt;- dat[in_train,]
  testClass &lt;- dat[-in_train,]
  
  
  
  set.seed(35)
  mod &lt;- train(Species ~ ., data = trainClass ,
               method = "rpart",
               metric = "F1",
               trControl = trainControl(summaryFunction = f1, 
                                    classProbs = TRUE))
  
  print(mod)
  </code></pre>
  
  <p>I coded a manual F1 score as well, with one input the confusion matrix: (I'm not sure if we can have a confusion matrix in "summaryFunction"</p>
  
  <pre><code>F1_score &lt;- function(mat, algoName){
  
  ##
  ## Compute F1-score
  ##
  
  
  # Remark: left column = prediction // top = real values
  recall &lt;- matrix(1:nrow(mat), ncol = nrow(mat))
  precision &lt;- matrix(1:nrow(mat), ncol = nrow(mat))
  F1_score &lt;- matrix(1:nrow(mat), ncol = nrow(mat))
  
  
  for(i in 1:nrow(mat)){
    recall[i] &lt;- mat[i,i]/rowSums(mat)[i]
    precision[i] &lt;- mat[i,i]/colSums(mat)[i]
  }
  
  for(i in 1:ncol(recall)){
     F1_score[i] &lt;- 2 * ( precision[i] * recall[i] ) / ( precision[i] + recall[i])
   }
  
   # We display the matrix labels
   colnames(F1_score) &lt;- colnames(mat)
   rownames(F1_score) &lt;- algoName
  
   # Display the F1_score for each class
   F1_score
  
   # Display the average F1_score
   mean(F1_score[1,])
  }
  </code></pre>