Machine Learning and Natural Language Processing : Project Initiation I am in the research phase of a long project and am willing to get some useful feedback from your side about the most appropriate project path to take.

Current situation:


A large team of so called production editors (PEs) is semi-automatically processing MS word files in order to do the copy-editing and layout.
This is currently semi-automatized with word macros where the PEs will apply different macros on different text paragraphs in order to set the appropriate style, position, font etc.
Still many actions are manual and depending on the human eye checks, resulting at the end many mistakes and an average quality


Plan


Start using machine learning, later maybe neuronal networks, statistics probability and artificial intelligence
The usage of machine learning (to start with) could be in the way that the system will learn the manual tasks done by the user (meaning the tool was not able to do those correctly in the automatized way) and then try to apply those in next similar situation
The term of similar situations is important here as this will imply natural language processing (NLP). The staff is dealing with word documents that could contain anything (within a similar layout)


The dilemma of the day is :) :

Where to start from in order to do some use case analyses? What I currently have:


More than 100k word files that are correctly done (except the user mistakes that may still lay around)
A list of rules that apply to the word documents describing the exact styles that needs to be used (non exhaustive)
Basic Machine Learning knowledge (starting with AI studying)
Starting with a small chunk of documents (~50) as a training set with a set of rules implements would be a good starting point. Train using a few algorithms and see if your training accuracy is coming out as acceptable or better than the human accuracy. 

Then slowly increase your training batch size and see the impact on the results.

Once you believe that the training set is a partially good representation of the dataset, go ahead do a test run. Then keep growing from there.