A few questions to understand a random forest blog I'm trying to understand a nice blog on the trade-off between sensitivity versus specificity with the random forest and logistic regression models. I have a few questions:

1) The blog used a 10 fold cross-validation in the ranger package in R (see the model mod_rf) and set the metric as ROC. So, is the final output (confusion matrix) we get there is for the one with the best ROC (AUC value) among the 10 validation sets? 

2) When I try to see the variable importance by varImp(mod_rf), it says the importance values are not available. Why is that? How can I get it?

3) The caret package in R allows upsampling to adjust for an imbalance in the data. They tried the logistic regression (see model sim_glm) to do upsampling and specified repeats = 2 to repeat the 10-fold cross-validation 2 times. How does it work? I'm not clear. Does it upsample females to create a 50-50 ratio of males and females "before" each fold of cross-validation? How would the process work for upsampling, repeats = 2 and 10-fold cross-validation in the case of a random forest?

4) If the AUC (from ROC curve) in my training data is about 10 percentage points less than the AUC from the test data, how should I explain that (this happened to my data)? I thought the training data would always show higher AUC than the test data because we used training data to build our model.   

I appreciate your responses.
I had a look at your questions and a brief look at the blog you posted. I will not go into detail and answer all your questions since it will take time and there should be one question. Nonetheless, I would like to give you some heads up on this and help you where possible.


When doing a confusion matrix is usually on a test set, you can see also where is used confusionMatrix(data = rparty.test, reference = testing$Sex). What is the usual is to have a training set, cross-validated to pick hyper-parameters and so forth and then test it on the test set. There is no rule just following a logical procedure. If these are new I would recommend diving into what are cross validations vs training set, test set and validation set.
Probably is talking about SMOTE upsampling (aka oversampling). This theory is for creating new artificial variables by creating synthetic ones (by combining features of the unprecedented class). Again I would recommend having a look there are many ways of dealing with an unbalanced dataset. Let's name a few: undersampling, oversampling, ensemble sampling (combing the previous) or you can tune certainly your model in order not to be fair towards a class and more.
Will not go into much detail about the ROC curve but I would recommend checking the AUC graphical plot instead of in this example and try to understand. Hint, the training dataset was balanced hence is reasonable a model would expect a balanced dataset for a test set instead is unbalanced.


All in all, I had a quick overview didn't go into depth of your blog and I didn't the find it very comprehensive plus uses confusing terms (as I remember), for instance, oversampling instead of upsampling or precision instead of specificity and recall for sensitivity. I wrote the answer as I empathise you as new in the area but next time please post only one question and make these sort of questions at the blog if applicable instead of here where the author directly can answer. Hope I helped you little bit, but you should also understand you opened a lot of topics.