Determine document novelty/similarity with the aid of Latent Dirichlet allocation (LDA) or Named Entities Given an index or database with a lot of (short) documents (~ 1 million), I am trying to do some kind of novelty detection for each newly incoming document.

I know that I have to compute the similarity of the new document with each document in the index. If the similarity is below a certain threshold, one can consider this document as novel. One common approach - which I want to do - is to use a Vector Space Model and compute the cosine similarity (e.g. by using Apache Lucene).

But this approach has two shortcomings: 1) it is computationally expensive and 2) it does not incorporate the semantics of documents and words respectively.

In order to overcome these shortcomings, my idea was to either use an LDA topic distribution or named entities to augment the Lucene index and the query (i.e. the document collection and each new document) with semantics.

Now, I am completely lost regarding the concrete implementation. I have already trained an LDA topic model using Mallet and I am also able to do Named Entity Recognition on the corpus. But I do not know how to use these topics and named entities in order to realise novelty detection. More specifically, I do not know how to use these features for index and query creation.

For example, is it already sufficient to store all named entities of one document as a separate field in the index, add certain weights (i.e. boost them) and use a MultiFieldQuery? I do not think that this already adds some kind of semantics to the similarity detection. The same applies to LDA topics: is it sufficient to add the topic probability of each term as a Payload and implement a new similarity score?

I would be very happy if you could provide some hints or even code snippets on how to incorporate LDA topics or named entities in Lucene for some kind of novelty detection or semantic similarity measure.

Thank you in advance.
Short answer: Topic Models and LDA aren't suited at all for novelty detection. I did some experiments with data that show a succession of topics in temporal order, and at the boundaries of the time interval one topic gets really high percentages in the document-topic matrix, i.e., the upcoming new trend (or the outrunning old trend) is not seen by LDA.

Topic Models are also not really suited for "online" processing—you read the whole corpus of documents once and than infer the topics.