r - rpart in text mining for classification of documents
rpart is a decision tree model and as such is very much interpretable.
You should visualize your decision tree. See the examples here, one of them is



Then for each given document you can follow the tree path and see 


what were the features on the way responsible for the classification 
what leaf your document arrived at and what is the percentage of correctly classified documents at these leaf.