Xgboost (classification problem) feature importance per input not for the model
A partial dependence plot might be what you are after: these plots show the relationship between the probability and the input variable. The mlr package in R takes care of this.

Discussion of Partial Dependence on XGBoost Git: https://github.com/dmlc/xgboost/issues/486

General Tutorial using mlr: https://mlr-org.github.io/mlr-tutorial/devel/html/partial_dependence/index.html