1. A method executed on one or more processors for automatically classifying documents based on topics and user inputs, comprising: receiving a set of documents which are classified as relating to the specific topic; producing an initial feature vector that corresponds to frequency of a term's occurrence in the set of documents; using the initial feature vector to classify another set of documents to produce an initial classified set of documents; receiving click information associated with a set of queries related to the specific topic, wherein the click information includes a click-through rate at which a query result is selected after being presented and a click duration indicating an amount of time during which the query result is accessed; using the click information to remove off-topic documents in the set of documents to obtain an updated set of documents, wherein a document is off-topic if the click-through rate or click duration associated with the document indicates the document is off-topic; determining an updated feature vector using the updated set of documents; and re-classifying the classified set of documents using the updated feature vector when the percentage of documents identified as off-topic exceeds a threshold which is greater than 0, otherwise retaining the initial classified set of documents.