Outlier detection for unbalanced classes
you need to distinguish between these cases:


Data Imbalance
Data Imbalance + Very few number of samples (minority class)
Severe Data Imbalance + Very few number of samples (minority class)


20:60   vs.  10:20    vs. 100:1000  vs. 10:100

and these cases:


similarities between different classes.
wide variations within the same class.


You need to understand to which of these cases your problem belong.

if you have very severe data imbalance + very few number of samples + wide variation within the majority class and similarities between different classes. regular oversampling or down sampling techniques will not help you as well as most of the synthetic oversampling techniques designed specifically to deal with the data imbalance but the assumption is to have enough number of samples.

Try to focus more on ensemble techniques that designed mainly to deal with data imbalance. 
SMOTE-Boost
RUSBoost
SMOTEBagging
IIIVote
EasyEnsemble