Deciding which samples the model will (probably) classify incorrectly
You might want to frame your problem as an uncertainty estimation problem.
The idea is that you want to evaluate how comfortable your model will be when making a prediction. If your model is not very comfortable with a prediction (even if gets classified as predic_proba = 0.99), then the uncertainty prediction should be high.
What @BrianSpiering is proposing is a way to calculate uncertainty with neural networks. This method is known as Monte Carlo Drop Out as a Bayesian estimation.
The idea is to apply drop out regularization when doing predictions and doing several times. This way your model predicts a probability distribution in which you can calculate several statistics as standard deviation. This will let you know how comfortable is your model doing such prediction.
This is just a method but there are several ways to estimate uncertainty.
There are some blogs and papers that might help you:

Uncertainty in Deep Learning. How To Measure?.Deep Learning Model Says: “sorry, I don’t know the answer”. That can be absolutely OK
Simple and Scalable Predictive Uncertainty
Estimation using Deep Ensembles: From Google DeepMind at NIPS