How to find "regions" with high purity I am applying a ML model (LGBM binary classifier) to data and would now like to identify the part of data where I have a low ratio of false-negatives (false-postives are not such a problem) and as much as possible true-negatives.

Background

The data I am classifying is from a system, that includes many complex rules that decide whether or not for some cases the company pays goodwill/warranty. In many of the cases this rule based system already finalizes the decision and initiates the payments. However there are cases, where this rule based system isn't able to finalize a decision but instead forwards the case along with a proposed decision to a manual decision process. It seems that this often also happens, because the rules (which are highly configurable) are not sharp enough or because the price-depending parts are not adjusted to inflation etc.

Now in this configuration it turns out, that the manual decision process very often just accepts the proposed decision. I am working on training a ML model (the LGBM classifier) to identify the cases, where the manual decision process leads to just accepting the proposed decision without changeing it.

Motivation

I am only interested in identifying cases where the proposed decision made by the rule system can just be taken over, because I'd like to reduce the amount of decisions that have to be checked manually.
My model currently gets a MAE of around 0.8 at the moment.

I'd now like to find a segment of the data, where the ratio of false negatives (the decisions where the model states, they need not change but in fact the descider changed them) is smallest (or below a certain threshold --> say 0.02).

Question

Can anybody suggest, how I could find such a region with a low ratio of false-negatives (false negatives in region by number of samples in the region)?
I thought about a simple decision tree, but would like to know if there are better options.

E.g. I suspect, that in case a leaf contains a high percentage of say 95% class 1 samples and 5% class 0 and some condition would allow to split it in two regions with 98% class 1 and 92% class 1 it wouldn't perform that split, since both would be classified as class 1 anyways, or am I wrong here?
A way to explore your dataset would be to use different techniques and see how they preform. The first approach I would consider is a tree based learner of modest complexity (or at least not boosted), like C4.5. It's possible (or perhaps likely) that won't perform as well as some more powerful approaches, but it has the added benefit of being relatively readable.

If you are familiar with a decision tree visualisation, you'll know that you can identify (leaf)nodes with high(er) impurity, and a large(r) population quite quickly. You can find an example of such a visualisation here. Those nodes can help you understand your domain, and possible how to improve performance (For instance by boosting, something LGBM already does for you). 

If you want to optimise the LGBM straight away, another thing that could help here is to cluster your dataset (kMeans could be a good starting point) and then calculate performance metrics per cluster. Possibly you could also consider clustering the scored items, including the classification, or even the evaluation (TP, FP, TN, FN). If used right that could help identify clusters of false negatives for instance, and the commonality they share. Disclaimer: I have never done this myself.

PS: Lot's of times I find it helpful to use KNN (over a subsample of the feature space) and Naive Bayes. They can quickly tell you something about the global correlations. Other golden oldies are feature importance rankers and a correlation plot.