title: Using R, is there a way to train and cross validate a random forest algorithm with the F1 score?
body: <p>I have data with class imbalance (the response variable has two classes, one of the classes is significantly more common than the other). Accuracy does not seem to be a good metric to train a model in this situation (I can get 99% accuracy and completely misclassify the minority class). I think that using the F1 score would be more beneficial.</p>
  <p>Has anyone ever tried using the F1 score as a training metric in R?
  I tried modifying the iris data set to make species as a binary variable and run random forest. Could someone please help me debug this?</p>
  <pre><code>library(caret)
  library(randomForest) 
  
  data(iris)
  
  iris$Species = ifelse(iris$Species == &quot;setosa&quot;, &quot;a&quot;, &quot;b&quot;) 
  
  iris$Species = as.factor(iris$Species) 
  
  f1 &lt;- function (data, lev = NULL, model = NULL) {
                  precision &lt;- posPredValue(data$pred, data$obs, positive = &quot;pass&quot;)
                  recall &lt;- sensitivity(data$pred, data$obs, postive = &quot;pass&quot;) 
                  f1_val &lt;- (2 * precision * recall) / (precision + recall) 
                  names(f1_val) &lt;- c(&quot;F1&quot;)
                  f1_val }
  
  
  train.control &lt;- trainControl(method = &quot;repeatedcv&quot;,
                                number = 10,
                                repeats = 3, 
                                classProbs = TRUE,
                                #sampling = &quot;smote&quot;, 
                                summaryFunction = f1,
                               search = &quot;grid&quot;)
  tune.grid &lt;- expand.grid(.mtry = seq(from = 1, to = 10, by = 1)) 
  
  random.forest.orig &lt;- train(Species ~ ., data = iris,
                              method = &quot;rf&quot;,
                              tuneGrid = tune.grid,
                              metric = &quot;F1&quot;,
                              trControl = train.control)
  </code></pre>
  <p>Gives the following error:</p>
  <pre><code>Something is wrong; all the F1 metric values are missing:
         F1     
   Min.   : NA  
   1st Qu.: NA  
   Median : NA  
   Mean   :NaN  
   3rd Qu.: NA  
   Max.   : NA  
   NA's   :10   
  Error: Stopping
  In addition: There were 50 or more warnings (use warnings() to see the first 50)
  5: stop(&quot;Stopping&quot;, call. = FALSE)
  4: train.default(x, y, weights = w, ...)
  3: train(x, y, weights = w, ...)
  2: train.formula(Species ~ ., data = iris, method = &quot;rf&quot;, tuneGrid = tune.grid, 
         metric = &quot;F1&quot;, trControl = train.control)
  1: train(Species ~ ., data = iris, method = &quot;rf&quot;, tuneGrid = tune.grid, 
         metric = &quot;F1&quot;, trControl = train.control)
  &gt; warnings()
  Warning messages:
  1: In randomForest.default(x, y, mtry = param$mtry, ...) :
    invalid mtry: reset to within valid range
  </code></pre>
  <p>Source: <a href="https://stackoverflow.com/questions/47820750/training-model-in-caret-using-f1-metric">Training Model in Caret Using F1 Metric</a></p>