Appropriate metric and approach for natural language generation for small sentences
I wouldn't say accuracy in the next word prediction is a good global metric. It would depend on the length of sentences. It's always difficult to predict the first word, cos you don't have any context. And having at least one word in the context it's easier. As long as your error rate is averaged among all predicted words, the accuracy could be higher if the lengths of sentences are higher. So the value of 0.5 doesn't tell much. The fact that you improved accuracy by 0.1 for instance on the same dataset means one method is better than another. Also, people measure perplexity and it's more sensitive to small changes of loss than accuracy. That's why I suggest you to measure perplexity. This metric also depends on lengths of sentences, that's why you should only compare 2 models trained on the same dataset.
What about GPT and Transformer, they were invented to solve one of the issues of LSTMs (they're not capable to memorize a very large context), that's why transformers would be better for long sentences, but they would also be better for short I think due to their attention mechanist, which has many useful properties.