In model validation (log regression), can AUC for the test sample be higher than that for the train sample? I have a relatively simple model (with 8'000 cases, 5 predictor variables) predicting a dihotomised outcome. It has an AUC of 0.82 (95%CI 0.82-0.83). Prediction in temporal (N=240) and external validation (N=170) datasets had AUCs of 0.85 (95%CI 0.81-0.88) and 0.76 (95% CI 0.68-0.84%). A reviewer told me it is impossible (can't happen, theoretically) for the temporal validation dataset to have higher AUC. Is this true?
This is just not true. Let's say that your future data is way easier to classify, then it can happen than temporal splits imply higher AUC. It often happens that future data is harder to classify because of shifts in the distributions of the variables, but if distributions are stable I don't see why not.