Get feature importance for each observation with XGBoost
I would suggest you're probably looking for something like one of these two packages:

https://github.com/slundberg/shap
https://github.com/marcotcr/lime   

See this notebook of lime's for example, which shows how you can use it to see why a specific sample in your data resulted in the prediction from the model:

https://marcotcr.github.io/lime/tutorials/Tutorial%20-%20continuous%20and%20categorical%20features.html