Model Predicts Badly on Real-World Data
You are using multiple frameworks (pandas, keras / tensorflow, nltk, reg, scikit-learn) and doing a lot of manually processing. It would be better if you used a single framework. A single framework would automatic many of the steps and appropriately handle training and prediction. The result would be creating the same one-hot encoding scheme at the token level.