Creating an easy but not trivial dataset I am working on the problem of automatic punctuation: given a stream of words, decide for each word whether there should be a punctuation mark after it (in future work I also want to distinguish between different punctuation marks, but currently it's a binary classification problem). 
My classifier uses a bidirectional LSTM whose output is fed into a multi-layer perceptron. The accuracy of the current classifier on the entire dataset is not good enough. In order to understand why, I want to create an "easier" dataset - a dataset on which even a simple (but not trivial) classifier would give good results. Then, I will make the dataset more and more difficult and see what improvements have to be made to the classifier. What is a good technique for creating synthetic datasets that are easier (but not trivial) to classify correctly?

In case it is relevant, here are some details on my system:


Each word is represented by an embedding vector of length 200.
The embedding vector is fed into a bidirectional LSTM with an output length of 50.
The LSTM output is fed into a multi-layer perceptron with two layers, the hidden layer has size 60.
The dataset has about 2000 paragraphs, each of which contains about 200 words.
Since the classes are imbalanced (only 20% of the words are punctuated), I check the system accuracy using Cohen's kappa coefficient - a number between -1 and 1. A random classifier has  kappa around 0. A good classifier has kappa above 0.7. My current classifier has kappa around 0.4 (average of 10-fold cross-validation), that is, better than random but not very good.
I train the classifier in 80 iterations. After each iteration I check the kappa of the model on a validation set. Finally I return the model with the highest kappa.
You might use a Markov chain for generating a simple enough dataset. You can estimate the probabilities of punctuation by scanning a large corpus of text (e.g. wikipedia), then you can use the model to generare fictious sentences with punctuation marks. You can set the order of the Markov chain so as to make the generated dataset increasingly closer to real-world text. The Markov chain should be defined on the set of words, however, so you have to carefully choose the vocabulary.