Categorizing Customer Emails I am working on a project for a company which needs to categorize customer e-mails regarding loans and insurance. The e-mails are labeled uniquely from set of 13 category labels. The number of records in the dataset are 3000. All e-mails are not in proper English or format, there are a lot of spelling mistakes and unwanted features as Indian Names, addresses etc.. The dataset is very imbalanced, the frequency of top 3 categories combines upto 2000 and there are 2 categories with frequencies ranging from 40-80.
I managed to pre-process the data and remove most of the unwanted stuff and spelling mistakes.

I have created a supervised learning model using LinearSVC which gave around 78% on validation dataset, after which I used XGBoost classifier which increased the accuracy to 80%. Both of these models use Tfidf Vectors with unigrams and bigrams of the pre-processed(stop words removed and stemmed text) as input.
I have tried SMOTE, TruncatedSVD and SelectKBest with chi2 but they decreased the accuracy.

In terms of Machine Learning Models I have used Multinomial Naive Bayes, Random Forests, Decision Tree Classifiers. I have even tried Feed Forward Neural Networks(MLP) and Recurrent Neural Networks for the classification but still XGBoost gives the best accuracy. I have also used Topical Modeling using LDA. I generated top words for each category and tried finding cosine similarities of each test set record with all the generated top words belonging to each category, still no improvement in accuracy.

So, my question is what kind of features should I look for? How do I extract more sophisticated features? My main objective is to increase the accuracy as the product is going to be an in-house software for the company. I am looking forward to context aware feature extraction or somehow engineer the features(add more or enhance them) to increase accuracy

PS: In the emails the customer ask for multiple actionable items, but still in the training set they are pre-classified as only one label corresponding to one of the actionable item asked in the e-mail. Plus, the data is also manually  wrongly classified by the previous workers. Please suggest and approach to capture the actionable items. I Have tried using class probabilities but they don't seem like giving the right answer, plus if the person has asked for only one actionable item then also the class probabilities will indicate other. I want to capture the actionable items as per asked by User. I have tried POS-tagging for this but due to bad quality of data, it is not giving good results.

PPS: The Email data which I have is in HTML format and has threads. I tried removing previous conversation threads before training but it resulted in bad accuracy. It is very strange that the more I try to refine the data by removing unwanted contents like disclaimers, auto-response emails, replies from customer-care representatives the more the accuracy falls.

Looking forward to get some insights on how to change my approach and specifically the features.

Thank You
(1) Data quality. The single best way to improve your accuracy. Garbage in garbage out. You already said your data is suspect. Some data was mis-classified; data only has single label, when multiple labels are possible. This is the biggie - this will improve your accuracy more than any other technique: improve the quality of your training data. One way to go about this. Recruit expert labellers (these could be workers in your own company who do this task, or they could be external people, e.g., mturkers who have been trained to do this labelling task). In general, the more labellers, the better. At a minimum I would have 3 labellers, label all n emails in your train/test set, using either majority label wins, or require all-three consensus depending on your confidence needs. You will want to measure your inter-rater reliability to ensure the integrity of your data using Cohen's kappa or similar. Importantly, this data will become your gold standard data. That is to say the average agreement upon labels provided by your original labellers is the ground truth to which your algorithm attempts to approach. 

Obviously, this approach to data quality is costly in terms of resources (time to label all the emails; cost of hiring mturkers, etc.). But depending upon your specific needs it needn't be... basically it depends upon the needed level of "expertise" to do the task. Sometimes, it is safe to consider you (the researcher/developer) as the "expert" (e.g., you yourself said that some emails were mis-classified... how do you know that? You must have some sort of expert knowledge.). However, using a single labeller for the task makes your data less reliable due to individual processing differences introducing a bias into the data.

(2) Including stop words. Yes, you read that correct including stop words. Grammatical function words (e.g., the, and, of, etc.) can often be highly indicative of specific text types or genres. It is premature to eliminate stop words without first ensuring that they are an uninformative feature. Similarly, try including punctuation, try including capitalization features. Time and time again, I see people blindly pre-processing text without empirically determining whether these features are significant predictors of class.

(3) Capture spelling variation. As the OP stated, the original texts have many "spelling mistakes." You can improve accuracy by reducing this variance - basically map variants to their canonical form before extracting your word features (e.g., "colour" -> "color"). I often rely on external corpora to rapidly create these spelling maps.

(4) Balance your training data - The OP mentions that his classes are severely imbalanced. Try over- and/or under-sampling to balance your classes.

(5) Describe your problem. It is also important to understand what the underlying purpose of your classifier. If your classifier is intended to run on historical data, perhaps to flag incorrectly human-classified emails, then you would probably want to keep auto-response and correspondence thread info. On the other hand, if the goal is to classify new customer emails so that they get properly routed, for example, then you probably want to remove these correspondence features from your training and test data.