classification algorithms for anomaly detection when there is class imbalance
Here is a pretty detailed summary on Handling Imbalanced Classes. Since you mentioned R, you might want to take a look at unbalanced package.