Oversampling possible improvement
The evaluation of an imbalanced dataset should be done with more than one single metric since you need to evaluate the performance on both the majority and the minority classes.

In the case that is mentioned the performance is evaluated only with recall (the true-positive rate). It could be that the model is now focusing on the oversampled minority class (especially if it's the new majority class) and having a weaker performance on the original majority class, and I assume that the precision or the false-negative rate were decreased.

There are several evaluation metrics that are considering the performance of both classes that are not sensitive to the classes imbalance - F1 score, Geometric mean, and the most common the ROC/PR AUC