Why doesn't LightGBM perform better than SVC (linear kernel)? [sentiment analysis]
Xgboost usually gives a higher accuracy(this is just an observation, not a fact). But the trick with xgboost is people dont know how to best use it. The most common mistake made using xgboost is not letting it train enough. Xgboost is an algorithm with overfits more than the other algorithms(well overfit is not the right term, it should be xgboost has high variance than the others). Do not tune the other parameters until you have found a converging nrounds for appropriate learning rate(0.01 or 0.1 or 0.3 etc). If you see that the cross val accuracy is much much lower than train accuracy then tune gamma(the second order derivative) first, then other parameters. If the cross val and train accuracy have a significant but not a lot difference(lets say train acc goes from 90 to 99 and cross val is stuck at 85 after significant iterations) then try reducing the max depth of trees. If the cross val and train accuracy are running side by side then just let the algorithm converge for a particular eta and then tune the other parameters(all except gamma).