Incorrect Text Classification, But Accurate Model. Do I Perform Manual Text Classification For A Data Set?
There can be two distinct reasons to use instances annotated with the gold-standard class, i.e. the true answer for the target application:


In order to perform proper evaluation your test set must contain the gold-standard labels. The principle of evaluation is to measure by how much the predictions deviate from the truth, but without the truth the performance that you obtain on the test set is meaningless for the task that you are doing.
In order to train a supervised or semi-supervised model, the training set must contain the gold-standard labels.  Semi-supervised methods offer some options to adapt a training set to a different task.


You can't rely on a model if you can't evaluate it at least on a small sample, so yes you probably need to manually annotate a subset of the data. It's only after that you can start thinking about how to improve performance.