Machine Learning Analysis for Redaction Purposes of Personally Identifying Information from Open Text Fields
NER

Basically the problem you have is Named Entity Recognition (NER).

This is a classic NLP problem and there are many machine learning approaches to it. Some recurring neural networks can be used, or the go-to method conditional random fields.

There are different techniques, usually in training, to overcome the issues (inconsistent reporting, nuanced terminology, abbreviations, spelling errors) that you mentioned.

Consideration

One important thing you need to consider is accuracy vs recall. You can choose to optimize one or the other, or both.

Accuracy = you try to redact something only if it really is some personal information.

Recall = you try to not miss any personal information.

Let say you have the sentence "Hello, my name is John, and my social security number is 123456789."

If you redact the sentence to: "Hello, -- ---- -- ----, --- -- ----- ------ ------ -- --------." You get really high recall (all personal information is gone)! But your accuracy is low, you redacted more than you should.

All this can be leveraged during prediction time. The probability score from your NER can make you optimize or the other.

Of course, this begs the questions: Why not redact EVERYTHING and so that 100% of personal information is not leaked?