Finding out why your model is doing better?
First, it makes sense that adding features will improve your performance, just make sure you do evaluation carefully and not overuse the same validation dataset (and if yes try to re evaluate it on an unseen independent different test set) to ensure you are not overfitted.
After that, you can use Shapley values in its aggregate look to see which features (or specific values in your encoding) impacted the model decision making