Why is KNN better at K-Fold Cross Validation than XGBoost or Random Forest? I've been running K-Fold cross validation multiple times for KNN, random forest and XGBoost.
KNN can complete sklearn's cross_val_score, so much faster consistently.
They all use the same preprocessed data, all have the same test/train split with a random state, etc.
When ran individually the timings are within 10% of each other.
When ran using sklearn's cross val score, KNN can do 50 k-folds in less than 10 minutes, when the other two take over an hour each.  (this is consistent, i've ran it 5 times for all 3 algorithms)
Mathematically, or maybe due to sklearn's code, is there a reason for this? (The dataset is 200,000 by 100 columns, at a 25% test/train split)
It's not that KNN is better at k-fold cross validation.
Its just that KNN doesn't do any training apart from storing a footprint of the training data within the model.
KNN's logic resides in its inference step i.e. the predict() call where it determines the k nearest neighbors for the newly supplied instance from previously supplied training data and makes a prediction for the label.
So it's likely that KNN is faster than most other models for small to medium sized datasets.