Feature engineering decrease my cross validation I'm currently working on a fraud detection data set. I'm evaluating my training data with a 10-skfold roc auc and an estimator of default param LightGBM. But, the problem is every time I try to create a new column by calculating a ratio of 2 column, my CV always drop even though the new column is theoretically will highlight a difference between a fraud or not.

The other problem is when I find 2 good (increasing CV) separate new feature by feature engineering when evaluating them independently, my CV actually decrease when I combine both of the feature on the same evaluation.

Is there anything wrong with my way of feature engineering? Right now my step is:
1. Create a new column based on other column
2. Evaluate CV with 10-skfold default param
3. If the CV is increased (relative to original data) then it's good feature else no.

Any help is appreciated.
You might be dealing with overfitting, since your model does not generalize that well. It should be noted that Light GBM is sensitive to overfitting in small datasets.

To solve this, you can check for features with correlation and drop some of them. Another solution can be the implementation of a feature selection method. Lastly, since GBM is a tree algorithm, you can try an early termination.

If you are dealing with highly imbalance data, you can consider using a stratified method rather than CV. In sklearn, there are StratifiedKFold and StratifiedShuffleSplit, which ensure that relative class frequencies are approximately preserved in both train and validation phases.