Machine Learning and Natural Language Processing : Project Initiation
Starting with a small chunk of documents (~50) as a training set with a set of rules implements would be a good starting point. Train using a few algorithms and see if your training accuracy is coming out as acceptable or better than the human accuracy. 

Then slowly increase your training batch size and see the impact on the results.

Once you believe that the training set is a partially good representation of the dataset, go ahead do a test run. Then keep growing from there.