NLP Feature creation from phrase matching I'm building a model to classify email content, to decide whether the email should lead to a JIRA ticket being "Raised" or "Not Raised". The problem I am having is the data is highly imbalanced with only around 11% being classed as "Raised". So far, the Random Forest classifier is providing the highest level of accuracy but the True Positive Rate/Recall is sitting at around 40% and I can't seem to increase upon this. I have been provided with a list of phrases that should they be contained in the email content, then in all likelihood a ticket needs raising. Looking for some tips as to the best method to create a new feature based on phrase matching? Has anyone any experience in the best methods for doing this?
The problem with imbalance is that the optimizer can get a very good score by declaring everything 'not raised'. You need to cheat with your training data by removing that incentive. I would suggest a training set that is balanced 50/50 between the classes. Your evaluation set can still be representative, which will give you a sense of how it'll generalize.