How to use "tree boosting" with a data-driven loss function
First some prior and known aclarations (that you probably already know).
Metric is what we want to optimize.
Optimization Loss is what the model optimizes.
Obviously, we would like the Metric and the optimization loss to be the same, but this is always not possible. How to deal with this?

Run the right model. Some models can optimize different loss functions. In the case of XGBoost you have two loss functions, the one of the decision tree and the one of the boosting.

Preprocess the target and optimize another metric, this will be for example transforming the  target to the logarithmic of the target and then in that space applying a known loss function

Optimize another loss function and metric and then post-process the predictions.

Write your own cost functions. For xgboost we implement a single function that takes predictions and target values and computes the first and second-order derivatives.

Optimize another metric and use early stopping.


The last one almost always works.
In general for complex algorithms Neural Networks tend to work better due to the flexibility of the loss functions (more than in normal ML).