Optimising for Brier objective function directly gives worse Brier score than optimising with custom objective - what does it tell me?
Brier score has known shortcomings for very rare or very frequent events.

Binary logistic regression objective function is relatively more robust than the Brier score to the rate of rare or frequent events.

It is possible that the difference in performance between the two objective functions is due to the event frequency in the data set.