How to find the residuals of a classification tree in xgboost
There are 2 methods to "continue a model training":

Training from another xgboost model

If you have a pre-trained xgboost model, you can continue training using the xgb_model parameter in xgb.train.


  xgb_model  a previously built model to continue the training from.
  Could be either an object of class xgb.Booster, or its raw data, or
  the name of a file with a previously saved model.


Training residual

Using xgboost to training a residual or prediction of a model, you can following the demo provided in xgboost:
https://github.com/dmlc/xgboost/blob/master/R-package/demo/boost_from_prediction.R

Simply speaking, you can use the "base_margin" element when preparing data.