# What formula is used to calculate perplexity in fitlda?

3 views (last 30 days)
Stephen Bruestle on 22 Jan 2019
Answered: Ilya on 13 Mar 2019
Many sources have different formulas. I want to make sure that I am referencing the correct formula.

Ilya on 13 Mar 2019
If you are asking about the 2nd output from the logp method, document log-probabilities are estimated using the Mean-Field Approximation described in the paper cited at the bottom of that doc page. Perplexity is then
exp(-sum(logprob)/Nwords)
where Nwords is the total word count across all documents.
If you are asking about perplexity displayed during training when you pass 'Verbose' to fitlda, those document log-probabilities are computed using current estimates of topic probabilities per document. The perplexity formula is the same as above. Because document log-probabilities are evaluated at the max likelihood estimates of topic probabilities per document, these document probabilities are overestimated and perplexity is therefore underestimated. This is done for speed. The MFA approach gives a more accurate estimate by integtrating over topic probabilities at the cost of longer runtime.
{"ft_lang_label":"__label__en","ft_lang_prob":0.9115088,"math_prob":0.78567314,"size":933,"snap":"2020-45-2020-50","text_gpt3_token_len":185,"char_repetition_ratio":0.17330463,"word_repetition_ratio":0.030075189,"special_character_ratio":0.17256163,"punctuation_ratio":0.07096774,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.99404955,"pos_list":[0],"im_url_duplicate_count":[null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2020-10-28T17:38:13Z\",\"WARC-Record-ID\":\"<urn:uuid:d2060d28-c4a0-41b0-a2b7-260f7a2bee00>\",\"Content-Length\":\"103387\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:fbd84bdc-f6c3-4f67-95a0-abf7fb013ce7>\",\"WARC-Concurrent-To\":\"<urn:uuid:39c7f42e-8235-4713-9119-e1181e09b059>\",\"WARC-IP-Address\":\"184.25.198.13\",\"WARC-Target-URI\":\"https://www.mathworks.com/matlabcentral/answers/440960-what-formula-is-used-to-calculate-perplexity-in-fitlda\",\"WARC-Payload-Digest\":\"sha1:IR4QK4DBDFBBXBO4B6ZMEB5EE2CZTIHP\",\"WARC-Block-Digest\":\"sha1:FKVMSL4P323KMD2A57ZOSVZWFKUXLX33\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2020/CC-MAIN-2020-45/CC-MAIN-2020-45_segments_1603107900200.97_warc_CC-MAIN-20201028162226-20201028192226-00017.warc.gz\"}"}