Classification when one class is other I am working on a litigation support application using the Enron corpus, which contains about 600,000 unique text documents.

In litigation, one is often concerned with whether a document is responsive or non-responsive. One produces responsive documents to the opposing side, unless they are privileged (e.g., attorney client communication).  

Here, I have sample sets of over 200 responsive and non-responsive documents.  

The challenge here is that the topic of the responsive documents is about one thing, whereas the non-responsive documents could be about any number of topics,  ranging from spam, to soccer practice, to business documents, etc.  The non-responsive is what I would call diluted.

I don't know what those non-responsive classes are up front, and there is no purpose or value to breaking them out up front. If a document is non-responsive, then it needs to be quickly (and cheaply) dismissed.

If the samples are random when the classification is applied to the corpus my customers expect the split between corpus responsive and non-responsive to be close to the split of the sample.

What is the best approach to classify responsive versus non-responsive in this situation?    

Below I briefly describe what I have tried. If there is a better approach, please share.


Using tf-idf, create an average vector for each document class (responsive and non-responsive).
Take the first 1000 terms (sorted by weight), such that each vector is the same length. 
Normalized the vectors.
Note: these two vectors are only a 0.03 cosine similarity to each other.
For each document in the responsive sample set, calculate the cosine similarity to the single average vector.
Note: the average is 0.06.  It is a small number, as documents in the sample set have around 40 terms, as compared to an average vector of 1000. 
In the non-responsive perform the same analysis (compare each non-responsive sample document to the non-responsive average vector).
In the non-responsive that same comparison is 0.03. Basically, the non-responsive average vector is diluted.


Based on this approach, I cannot really conclude if document has a higher cosine similarity to responsive compared to non-responsive then it is responsive.  By that measure 1/3 of the documents in the non-responsive sample set would be classified as responsive.    

The non-responsive needs to be handicapped.  In this approach, how do I handicap it?  Is there another approach that would accomplish the same thing?
As a first step you should try logistic regression on your tf-idf vectors.  It is simple to implement and would provide a good baseline for comparison.  You can find an implementation in whatever language you're using.  You could also try some kind of (perhaps supervised) topic modeling to create a better feature space, but that would be more involved.