Algorithms and tools for ranking text as a job description
If I understand the problem correctly, there are several approaches that can be taken.

We'll begin with the least deep learning oriented solution first and slowly move into the spectrum of Deep Learning. This choice is primarily for cost and difficulty reasons. The amount of bugs that you can find with a deep learning solution can be very difficult to understand in certain situations and that results in slower prototyping and ultimately slower time to production.

At the very base, your problem deals with ranking the jobs with respect to how relevant they are to a particular job. The literature for ranking problems is vast and I will suffix my answer with links to these papers or studies. I will also say that instead of dealing with this as a ranking problem there may be another way of going about it. However, this is hinged on the type of data you possess. If you know that each query is certainly looking for a particular type of job. Then, you can break this down into a multi-task problem, where one of the tasks is figuring out whether there is a vacancy or not and the second is figuring out if the vacancy relates in any way to the search query.

Solution 1: Take the dataset and handcraft a set of features that you think are relevant to the process of classifying the output. To do this effectively, you should know certain core features that you think relate the input features to the final label (here I use input features to represent the features that you think represent your features well. For example, some features that may be relevant is information about the person or entity posting the title (in stage 2), if they are a well known company in the domain (this can be accesses through some intelligent web scraping and text retrieval methods). Then, it is possibly more likely for it to be a posting, right? Let us assume now that you have constructed this set of features. Now, for each of the set of features, you have a corresponding label to whether there is a vacancy or not. To turn this approach into a ranking problem is very simple. All you have to do is observe the confidence (the softmax probability is good enough) and the one which has a higher probability is ranked higher). This solution is an easy one to accomplish using even simple feed forward neural networks. You would also benefit from testing out simple classifiers like SVMs or Random Forests. It must also be remembered that the actual sentence must be embedded into some numeric space, like a sequence of vectors in $R^n$. This solution does not detail all of the explicit steps but provides a general framework with which you should be able to attack the problem.

Drawbacks The above method requires a lot of time spent in handcrafting the feature set and there is going to be a lot of iteration because the feature set is never going to be perfect. There is always going to be something that you possibly missed out. Secondly training a good word embedding is not a very easy task. You may have to spend a lot of time getting that running, especially if you want the embedding to be somehow related to your words.

Solution 2 This solution begins to touch the fence of Deep Learning. This solution is a comparison of the attention encodings within the source utterance and the target utterance. The source utterance is, a phrase like "need janitor". Now you should train an attention model over the sequence of words and observe what is the word(s) being given the maximum attention to. Once you know this, the problem becomes a question of consolidating the attentions in some meaningful manner and comparing the distance between the source and target through some metric space. So you should certainly pay attention to attention based models (haha!). Again, there are going to be hiccups in the process of figuring out how much attention actually affects the output and whether it is a good model for your data. Distances between attention will return a vector and you can consider the $\mathcal{L}_2$ norm or any other map from $\mathbb{R^n} \rightarrow \mathbb{R}$.

Drawbacks Attention distances are a good metric, however, they are extremely data centric. For example if the ads are very long, it may be true that the model does not pick up the appropriate words to attend to.

Now we get to some resources that may be useful to go through in order to understand exactly how the different components of the answer work.


https://en.wikipedia.org/wiki/Learning_to_rank
http://www.shivani-agarwal.net/Events/SDM-10-Tutorial/sdm10-tutorial.pdf
https://towardsdatascience.com/learning-to-rank-with-python-scikit-learn-327a5cfd81f
http://times.cs.uiuc.edu/course/598f16/l2r.pdf
https://www.stat.uchicago.edu/~lekheng/meetings/mathofranking/slides/agarwal.pdf


Now for the posts about attention:


http://www.wildml.com/2016/01/attention-and-memory-in-deep-learning-and-nlp/
http://akosiorek.github.io/ml/2017/10/14/visual-attention.html
https://medium.com/syncedreview/a-brief-overview-of-attention-mechanism-13c578ba9129


Finally, my last component of the answer will deal with evaluating exactly how well you have done using metrics. I have added some metrics above in my Answer, however there is another metric which in particular represents the similarity, syntactically between the input query and the output query.


https://en.wikipedia.org/wiki/BLEU