Semi-supervised anomaly detection
If you use the anomaly detector to label the data directly, there is no way the  supervised step that follows can be better than that. One can of course go in an "adjust" the labels afterwards, but there is risk of being biased by the pre-existing labels if a human sees it up front.

Instead of sampling data to label randomly, you could sample weighted based on the anomaly score. This has two effects 1) reduces class imbalance 2) focuses labeling effort on likely anomalies

This assumes a well-tuned anomaly detector though. Which there is no good way to do without a validation set. That dataset should preferably be sampled randomly, to avoid bias.