Neural network training with known erroneous data
If you want to identify and remove problematic data, then this is much better done before training.

You might have some luck with your approach, but you have no guarantees that the neural network will help you isolate the problem entries based on error values, if it has been trained on the "good" and "bad" entries at the same time. It depends on the nature of the problem data. You take a significant risk that the model will fit to the problem data well enough to cause you to reject good data.

You should instead try to think of a way to identify the bad data more directly, and prior to training for the predictive task.

Here's one idea: Are you able to identify enough good and bad data manually - enough to train and test a classifier? Then label some of the data "good" or "bad", train a classifier and test it - the test should be on some held out labeled values, to help you assess accuracy. If your classifier has a good accuracy, you can have some confidence to use it to filter the remainder of your data, and only use the "good" classified data for your original training goal.