Balancing XGboost still skews towards the majority class
That is completely normal. You should remember that the model will basically learn a statistical function given by your data (Intuitively), and since your data is skewed, it will learn by the majority class.

To overcome that, you can treat the imbalance characteristic of data set using two types of approaches: sampling and cost-sensitive learning. Basically, in sampling techniques we have undersampling and oversampling, that is, we can reduce our data set to balance data or increase it with artificial methods. Some examples:


Random Oversampling: essentially, this technique increases minority class by randomly sampling it.
SMOTE: random oversampling has a overfitting problem as we are just replicating data considerably. To overcome that, smote works, in a simplified way, interpolating points on the feature space for nearest neighbors in minority classes.


For undersampling techniques:


Random Undersampling: this method simple undersample the majority class. We can loss information, but if points are close, it can give good results
NearMiss: In a simplified way, this model selects the closest points to the minority class.


In cost-sensitive learning techniques, we use cost matrix 'punishing' models that give wrongs score to a specific class. For example, let's say you want to give better results for class C1, you can create a matrix where the misclassification of C1 is higher than other classes.

In python we have two excellents libraries to deal with this problems: 


https://imbalanced-learn.readthedocs.io/en/stable/api.html
https://wwwen.uni.lu/snt/research/sigcom/computer_vision_lab/costcla_a_cost_sensitive_classification_library_in_python


They have excellent tutorials on the topic so you can get a deeper overview of methods discussed and others ones. I hope it helps.