How to identify new job descriptions/postings from a set of documents when I have a set of already labeled job descriptions/postings Suppose I have a set of already labeled documents -- some of them are job descriptions/postings (these are documents of interest), and some of them are not. I wonder what kind of method would allow me to build a model that can generalize to new data, specifically new job descriptions that may be very different from the already labeled job descriptions.

What I have done so far (in Python):

(1) Trial 1: I tried the classic bag-of-words + TfidfVectorizer + binary classification (LogisticRegression specifcally) approach. With parameter tuning based on the train + valid sets, precision and recall on the test set could reach over 98%. However, after I built the model, I collected a new set of documents (again, some are job postings and others are not), and asked annotators to label these new data. I then used the model to classify these new docs. Perhaps expectedly, the precision and recall dropped pretty drastically to, ~65% and ~40%.

(2) Trial 2: I thought perhaps using tfidf alone might cause some overfitting, so I tried applying TruncatedSVD (with 100 components) on the TfidfVectorizer (bag-of-words + TfidfVectorizer + TruncatedSVD + binary classification).  Again, I tuned the parameters using the train and valid sets, and the precision and recall were similarly high on the test set. When I applied this new model to the newly collected data, the recall improved a bit, from 40% to 50%, but the precision dropped a bit, to 58%.

So my question is: is there a way to leverage the initial set of coded documents to build a kind of model that can generalize better to new data set. My specific interest is identifying job descriptions/postings. It doesn't have to be a binary classification task since the non-job-descriptions can be much more varied than job descriptions/postings
You should at least try pre-trained embedding vectors. TfidVectorizer is particularly sensitive to out-of-vocabulary words, which are likely to appear if you're trying transfer learning to a new domain. The GloVe embeddings [1] have a dictionary of 400k vectors, unless you're working with documents from a technical domain they should provide some improvement. Also, consider to perform some extra feature engineering, words by themselves do not provide much information. Why not trying adding n-grams &amp; dependencies extraction, entity recognition, etc. as preprocessing steps?

[1] http://nlp.stanford.edu/projects/glove/