How to score arbitrary sentences based on how likely they are to occur in the real world? According to article about LSTM here, I know that:


  it allows us to score arbitrary sentences based on how likely they are to occur in the real world. This gives us a measure of grammatical and semantic correctness. Such models are typically used as part of Machine Translation systems. 


But, it seems that this article doesn't point out how to compute the score with LSTM.

Any way to compute the score?
Typically you would use a perplexity value. For example, if your LSTM model is word-based and you have a sentence $[x_1, x_2 . . . x_N]$, and your model predicts the words that appear in that sentence with probability $p(x_i|x_0..x_{i-1})$ (where $x_0$ is a "start token" or whatever you use to start your RNN prediction sequence). Then you might quote a per-word perplexity for that sentence under your model as

$- \frac{1}{N}\sum_{i=1}^N log_2(p(x_i|x_0..x_{i-1}))$

Using an LSTM to predict consecutive words, it is practical to construct an array of probabilities $[p_1, p_2 . . . p_N]$ by running the network on the sentence and noting the probabilities for the correct matching word - i.e. $p_i = $ the predicted probability of the correct class $x_i$ at each step, which simplifies the expression:

$- \frac{1}{N}\sum_{i=1}^N log_2(p_i)$