Is there a good alternative to XGBoost for learn to rank?
You asked for alternatives - also consider: LightGBM, CatBoost or RankLib.
How big is your dataset?  If you are hitting memory problems just creating the DMatrix its possible the other libraries will have similar problems.
What are you loading your dataset from?  Perhaps creating the DMatrix from a numpy array is trying to fit two copies of your data in memory at the same time?  Perhaps you could divide your dataset into smaller chunks and append them to the DMatrix to avoid having them all in memory at once?