Undersampling for credit card fraud detection before or after Train/Test Split I have a credit card dataset with 98% transactions are Non-Fraud and 2% are fraud.
I have been trying to undersample the majotrity class before train and test split and get very good recall and precision on the test set.
When I do the undersampling only on training set and test on the independent set I get a very poor precision but the same recall!
My question is :

Should I undersample before splitting into train and test , will this mess with the distribution of the dataset and not be representative of the real world?

Or does the above logic only apply when oversampling?
Do the undersampling after the train/test split and only in the train split: you want to somehow &quot;weigth&quot; your learning algorithm in order to prevent it to be biased towards the majority class, indeed, other techniques can be applied, and always on the train split, what eventually is the data for the learning algorithm. But... your test set should represent the real data distribution. In this case, you will asses if whatever technique you have applied to prevent a biased algorithm towards the majority class has not eventually affected its performance on it.
Some other approaches for this problem (fraud vs non-fraud):

Anomally detection
One-class classifies: One Class SVM