Our attempt at language pre-training fell short of our expectations in all but one tested dataset. Our pre-training was unsuccessful in improving accuracy, even when applied to networks larger than those reported. 
 Question: Does pre-training on general text corpus improve performance?
No