LightGBM results differently depending on the order of the data I have two datasets A and B which are exactly the same in terms of the number of columns, name of columns and the values. The only difference is the order of those columns. I then train the LightGBM model on each of the two datasets with the following steps


Divide each dataset into training and testing (use the same random seed and ratio for both A and B)
Leave the hyperparameters as pretty much default
Set a random state as a fixed number (for reproduction)
Tune the learning_rate using a Grid Search
Train a LightGBM model on the training set and test it on the
testing set
Learning rate with the best performance on the testing set will be
chosen


The output models on the two datasets are very different, which makes me thinks that the order of columns does affect the performance of the model training using LightGBM.

Do you know why this is the case?
A possible explanation is this:

When the order of the columns differ, there is a little difference in the procedure.

What LightGBM, XGBoost, CatBoost, amongst other do is to select different columns from the features in your dataset in every step in the training.

The selections of these columns is done randomly: Let's say your dataset has 20 columns. The root node selects the features 1st, 3rd and 18th, on both datasets the 1st, 3rd and 18th features are different in both possible datasets. This is repeatedly done and in every step there is randomness affecting your ultimate result.