Model Predicts Badly on Real-World Data I know this is probably a really easy fix, but I've been stuck on this for a while. I'm a noobie to both machine learning and programming in general.  My data is from this dataset. https://www.kaggle.com/ozlerhakan/spam-or-not-spam-dataset
I made a model that can predict whether or not an email is spam email, and it does so reasonably well, getting an 84% accuracy with the validation data. But when I try to make it predict on a new string, it seems to be predicting on a character-by-character basis. Below is the code that I used to preprocess the training(and testing) data. I would've copy-pasted the required code, but when I look at the draft, it looks terrible. So here's the link to my notebook. https://colab.research.google.com/drive/1EnoNFR5CFi85hOrW8ihx66QWb6sToKAL?usp=sharing
I tried to debug the function by putting various print statements all over. I've come to the conclusion that the problem spot is when I try to one-hot encode the words. I need those words to be one-hot encoded because that is how I trained the model.  But I don't know how to fix that. I would really appreciate some help with this. Thank you in advance.
You are using multiple frameworks (pandas, keras / tensorflow, nltk, reg, scikit-learn) and doing a lot of manually processing. It would be better if you used a single framework. A single framework would automatic many of the steps and appropriately handle training and prediction. The result would be creating the same one-hot encoding scheme at the token level.