Overfitting due to features correlating with training set generation rules
I would say that there are no "problems" in the sense what is happening is to be expected.

First of all, here is a key ML reminder which somehow often gets lost:


Performing well on the test set is pointless if that test set is not representative (i.e. is not similar to unlabeled instances)
Adding more training data does not help if that training data isn't covering new situations


You say you created your training set using string matching rules, I assume that these rules are similar to " "mortgage repayment" was a "housing" observation." as you pointed out in the comments.

Since your model considers bigrams as input, it is not surprising that it found a way to reverse-engineer which string matching rules you used.

To improve your model, I would look at what mistakes your model currently makes on unlabeled data, this should provide you with training instances that do not fit your string matching rules.