# Compare Models. LOCCV implementation in R

I want to compare three models, one linear-regression-model, one regression-tree-model (from rpart) and one MARS-model (from mda package).

I want to compare the models using a leave one out cross validation using the mean square error and MAPE. I have the following implementation in R:

library(data.table)
library(rpart)
library(mda)

#Load Sample-Data
data(trees)

#The following models should be compared:
# lm(Volume~Girth+Height, data=trees)
# rpart(Volume~Girth+Height, data=trees)
# mars(trees[,-3], trees)

LOOCV<-function(modelCall) {
unlist(sapply(seq(1,nrow(trees)), function(i) {
training=trees[-i,]
test=trees[i,]

fit=eval(modelCall)
testValue = predict(fit, test[1:2])

test-testValue
}))
}

LOOCV_MSE<-function(modelCall) {
sum(LOOCV(modelCall)^2)/nrow(trees)
}

LOOCV_RMSE<-function(modelCall) {
sqrt(LOOCV_MSE(modelCall))
}

LOOCV_MAPE<-function(modelCall) {
sum(abs(LOOCV(modelCall)/sapply(seq(1, nrow(trees)), function(i) {trees[i,3]})))/nrow(trees)*100
}

cat("Cross-Validation Metrics:\n")
cat("-------------------------\n")
cat("LOOCV MSE for LM:", LOOCV_MSE(quote(lm(Volume~Girth+Height, data=training))),"\n")
cat("LOOCV MSE for CART:", LOOCV_MSE(quote(rpart(Volume~Girth+Height, data=training))),"\n")
cat("LOOCV MSE for MARS:", LOOCV_MSE(quote(mars(training[,-3], training))),"\n")
cat("\n")

cat("LOOCV RMSE for LM:", LOOCV_RMSE(quote(lm(Volume~Girth+Height, data=training))),"\n")
cat("LOOCV RMSE for CART:", LOOCV_RMSE(quote(rpart(Volume~Girth+Height, data=training))),"\n")
cat("LOOCV RMSE for MARS:", LOOCV_RMSE(quote(mars(training[,-3], training))),"\n")
cat("\n")

cat("LOOCV MAPE for LM:", LOOCV_MAPE(quote(lm(Volume~Girth+Height, data=training))),"\n")
cat("LOOCV MAPE for CART:", LOOCV_MAPE(quote(rpart(Volume~Girth+Height, data=training))),"\n")
cat("LOOCV MAPE for MARS:", LOOCV_MAPE(quote(mars(training[,-3], training))),"\n")


Outputs:

Cross-Validation Metrics:
-------------------------
LOOCV MSE for LM: 18.15783
LOOCV MSE for CART: 69.83769
LOOCV MSE for MARS: 13.72282

LOOCV RMSE for LM: 4.2612
LOOCV RMSE for CART: 8.356895
LOOCV RMSE for MARS: 3.704432

LOOCV MAPE for LM: 14.6114
LOOCV MAPE for CART: 23.51401
LOOCV MAPE for MARS: 10.00316


Does this implementation make sense? When whould using MSE on the errors make sense? When would I use MAPE/SMAPE instead? I already read "Metric to compare models?" and the conclusion there was it depends, can someone explain this further. On what does it depend?

My data is not a time series, it is more like the tree example data.

## 1 Answer

Why re-invent the wheel? R already has many libraries that implement cross-validation and calculate RMSE, MSE, MAPE, etc:

library(caret)
library(forecast)
library(rpart)
library(mda)

#Load Sample-Data
data(trees)

#Custom Summary Function for Cross-Validation
customSummary <- function (data, lev = NULL, model = NULL) {
stats1 <- postResample(data[, "pred"], data[, "obs"])
stats2 <- accuracy(data[, "pred"], data[, "obs"])
c(stats1, stats2)
}

#Choose sampling method and sumamry function
myControl <- trainControl(method='LOOCV', summary=customSummary)

#Run Models
model_LM <- train(Volume~Girth+Height, data=trees, method='lm',
trControl=myControl)
model_CART <- train(Volume~Girth+Height, data=trees, method='rpart',
trControl=myControl)
model_MARS <- train(Volume~Girth+Height, data=trees, method='earth',
trControl=myControl)

#Compare models
model_LM
model_CART
model_MARS


Choosing which statistics to use is up to you, and depends on the particulars of your problem. I usually use MAE as a default, but it pays to think about what the actual error cost in your problem.
{"ft_lang_label":"__label__en","ft_lang_prob":0.72662246,"math_prob":0.97745866,"size":2523,"snap":"2021-21-2021-25","text_gpt3_token_len":834,"char_repetition_ratio":0.1814212,"word_repetition_ratio":0.0,"special_character_ratio":0.31589377,"punctuation_ratio":0.18163265,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.9961257,"pos_list":[0],"im_url_duplicate_count":[null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2021-06-24T09:11:35Z\",\"WARC-Record-ID\":\"<urn:uuid:70fa21ab-46f0-4e95-82ea-ae72419d39a5>\",\"Content-Length\":\"164074\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:da8d882c-d011-4bf7-87a5-d58162b9bdfb>\",\"WARC-Concurrent-To\":\"<urn:uuid:93e2e984-1457-41d4-a94b-dd6caaf57204>\",\"WARC-IP-Address\":\"151.101.1.69\",\"WARC-Target-URI\":\"https://stats.stackexchange.com/questions/27351/compare-models-loccv-implementation-in-r\",\"WARC-Payload-Digest\":\"sha1:Q5PYCQ2XDJWUGWBZTGRIBO47ZOCFAGUC\",\"WARC-Block-Digest\":\"sha1:2KWSWUKDXM4J7ZY3DAIDS4JWKZIBWORM\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2021/CC-MAIN-2021-25/CC-MAIN-2021-25_segments_1623488552937.93_warc_CC-MAIN-20210624075940-20210624105940-00167.warc.gz\"}"}