Improving accuracy of Text Classification I am working on a text classification problem, the objective is to classify news articles to their corresponding categories, but in this case the categories are not very broad like, politics, sports, economics, etc., but are very closely related and in some cases even partially overlapping. This is a single category classification problem and not multi-class classification. Below is the details of the method I used.

Data Preparation -


Broke the documents in list of words.
Removed stop words, punctuations.
Performed stemming.
Replaced numerical values with '#num#' to reduce vocabulary size.
Transformed the documents into TF-IDF vectors.
Sorted all words based on their TF-IDF value and selected the top 20K words, these will be used a feature list for the classification algorithm.
Used SVM.


I have 4,500 categorized documents with 17 categories, and I used 80:20 ration for training and test dataset. I used Sklearn python library.

The best classification accuracy I have managed to get is 61% and I need it to be at least 85%.

Any help on how I can improve the accuracy would be greatly appreciated. Thanks a lot. Please let me know if you need any more details.
First of all good job done in processing the data and coming up with your base model. I would suggest few things that you can try:


Improve your model my adding bigrams and tri-grams as features. 
Try doing some topic modelling like latent Dirichlet allocation or Probabilistic latent Semantic Analysis for the corpus using a specified number of topics - say 20.  You would get a vector of 20 probabilities corresponding to the 20 topics for each document. You could use that vector as input for your classification or use it as additional features on top of what you already have from your base model enhanced with bigrams and trigrams. 
Another thing I would say is try using a tree based classifier ensemble to capture non-linearity and interactions between the features. Either of random forest or gradient boosting would be fine. In gradient boosting you can use  xgboost as its a pretty good package that gives good classification. 
If you are familiar with Deep Learning you can give a try with a Recurrent Neural network architecture(mostly the LSTM versions).  


Hope this helps and let me know if this improves your classification accuracy.