How to decide on using xgboost with imputation or without it and keeping missing values?
The first question about missing data is always why is it missing?
Have you checked or know why the data is missing and whether it is MAR, MCAR or not missing at random?
If your data is MCAR imputation is generally fine and your lower test metric might simply indicate a suboptimal imputation strategy. In this case you could try MICE or similar more advanced imputations than simple median imputation.
That not imputing missing values actually improves your prediction might indicate that your data isn't missing completely at random however. In this case coding missing values might improve your performance and therefore be the best course of action.
As described besides simply looking at the performance metric of a properly validated test set also try to understand why data is missing and what that would mean.