# Support vector regression with one large outlier

I have a question regarding support vector regression, best summarized by the chart below on simulated data of a linear function with a bit of noise. In essence, why does increasing epsilon rotate the fitted line away from the outlier?

Note: the simulated data is the red dots. Green line is a naive lm fit, blue lines are svr predictions for varying values of epsilon.

Motivation: trying to understand SVR in the face of a single large outlier. Code here:

rm(list = ls())
library('e1071')
library('tidyverse')

n <- 1000
set.seed(250)
x <- 1:100
y <- 5*x + rnorm(length(x), 0, 10)
# y <- 500 #+ rnorm(length(x), 0, 10)
df <- data.frame(x, y)
df[nrow(df), ] <- c(100, 1000)  # add outlier at last x value
df <- df %>% arrange(x)

res <- list()  # blank list for results
epsilon <- seq(0,2.5,0.2)  # epsilon values to consider
# fit svr models
for (i in 1:length(epsilon)) {
eps <- epsilon[i]
svrModel <- svm(y ~ x, df, type = "eps-regression", kernel = "linear", epsilon = eps)
svrTest <- predict(svrModel, df)
res[[i]] <- data.frame(eps = eps, x = df$x, pred = svrTest) } # lm for comparison lmModel <- lm(y ~ x, df) predLM <- predict(lmModel, df) dfLM <- data.frame(x = df$x, y = predLM)

resTidy <- bind_rows(res)  # tidy data

resTidy %>%
ggplot(aes(x = x, y = pred, group = eps, color = eps)) +
geom_line(aes(colour = eps), size = 1.2) +
geom_point(data = df, aes(x = x, y = y), alpha = 1, col = 'red') +
geom_line(data = dfLM, aes(x = x, y = predLM), colour = 'green', linetype = 2, size = 1.2)


In particular the second link listed below (part way down) says, based on the Smola paper (link 3):

In SVR we want to maximize the prediction error in a defined precision, epsilon for better generalization. Here if we minimize the prediction error instead of maximize, the prediction result on unknown data is more likely to be over-fitted.

Thoughts greatly appreciated...finding SVR slightly unintuitive vs. SVM for classification.

P.S. Links that may be useful (I don't have enough reputation points to post all as hyperlinks) - remove the @@ which I inserted to be allowed to post them:

1. https:@@//stats.stackexchange.com/questions/13194/support-vector-machines-and-regression
2. https:@@//stats.stackexchange.com/questions/5945/understanding-svm-regression-objective-function-and-flatness
3. https:@@//alex.smola.org/papers/2004/SmoSch04.pdf
{"ft_lang_label":"__label__en","ft_lang_prob":0.7021328,"math_prob":0.9895609,"size":2337,"snap":"2023-40-2023-50","text_gpt3_token_len":666,"char_repetition_ratio":0.08915559,"word_repetition_ratio":0.0054054055,"special_character_ratio":0.31236628,"punctuation_ratio":0.16594827,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.9996693,"pos_list":[0,1,2],"im_url_duplicate_count":[null,3,null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2023-11-30T12:28:51Z\",\"WARC-Record-ID\":\"<urn:uuid:a9dde6b9-00c1-41c0-8bcb-ab628353e1cc>\",\"Content-Length\":\"159209\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:56270499-e287-4060-8ff7-bf26b8d6e5a9>\",\"WARC-Concurrent-To\":\"<urn:uuid:c77ac582-ab2f-48aa-9ebf-68b7aa6c2e56>\",\"WARC-IP-Address\":\"104.18.43.226\",\"WARC-Target-URI\":\"https://stats.stackexchange.com/questions/291583/support-vector-regression-with-one-large-outlier\",\"WARC-Payload-Digest\":\"sha1:ARGH4FCYLLHPIRCD2BI6N3FUFBT6YZPE\",\"WARC-Block-Digest\":\"sha1:K66WCXLAGOMSEBRYLPBFSWMY6WHBTHLD\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2023/CC-MAIN-2023-50/CC-MAIN-2023-50_segments_1700679100184.3_warc_CC-MAIN-20231130094531-20231130124531-00559.warc.gz\"}"}