Posted on 1 Comment on Some Details on Running xgboost

## Some Details on Running xgboost

While reading Dr. Nina Zumel’s excellent note on bias in common ensemble methods, I ran the examples to see the effects she described (and I think it is very important that she is establishing the issue, prior to discussing mitigation).

In doing that I ran into one more avoidable but strange issue in using xgboost: when run for a small number of rounds it at first appears that xgboost doesn’t get the unconditional average or grand average right (let alone the conditional averages Nina was working with)!

Let’s take a look at that by running a trivial example in R.

Posted on

## (or: how to correctly use `xgboost` from `R`)

`R` has "one-hot" encoding hidden in most of its modeling paths. Asking an `R` user where one-hot encoding is used is like asking a fish where there is water; they can’t point to it as it is everywhere.

For example we can see evidence of one-hot encoding in the variable names chosen by a linear regression:

``````dTrain <-  data.frame(x= c('a','b','b', 'c'),
y= c(1, 2, 1, 2))
summary(lm(y~x, data= dTrain))``````
``````##
## Call:
## lm(formula = y ~ x, data = dTrain)
##
## Residuals:
##          1          2          3          4
## -2.914e-16  5.000e-01 -5.000e-01  2.637e-16
##
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)
## (Intercept)   1.0000     0.7071   1.414    0.392
## xb            0.5000     0.8660   0.577    0.667
## xc            1.0000     1.0000   1.000    0.500
##
## Residual standard error: 0.7071 on 1 degrees of freedom
## Multiple R-squared:    0.5,  Adjusted R-squared:   -0.5
## F-statistic:   0.5 on 2 and 1 DF,  p-value: 0.7071``````
{"ft_lang_label":"__label__en","ft_lang_prob":0.8145283,"math_prob":0.9178692,"size":1581,"snap":"2019-51-2020-05","text_gpt3_token_len":480,"char_repetition_ratio":0.083703235,"word_repetition_ratio":0.0,"special_character_ratio":0.33523086,"punctuation_ratio":0.15492958,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.95261735,"pos_list":[0],"im_url_duplicate_count":[null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2019-12-05T16:09:44Z\",\"WARC-Record-ID\":\"<urn:uuid:7958bf7c-2082-42b5-a282-6fec507807d8>\",\"Content-Length\":\"60051\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:5e98d99e-6536-4b3c-b719-c8b45d06e2bb>\",\"WARC-Concurrent-To\":\"<urn:uuid:bc4fbcad-5a89-4721-b9fc-6ccd465c3071>\",\"WARC-IP-Address\":\"98.129.229.190\",\"WARC-Target-URI\":\"http://www.win-vector.com/blog/tag/xgboost/\",\"WARC-Payload-Digest\":\"sha1:BYDXHUHACW2JWPRCEX5O3S3YNPJ2NEKB\",\"WARC-Block-Digest\":\"sha1:GATACR3QOPZOULVMNOUYLJR2UUG2YNPK\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2019/CC-MAIN-2019-51/CC-MAIN-2019-51_segments_1575540481076.11_warc_CC-MAIN-20191205141605-20191205165605-00173.warc.gz\"}"}