# Definition of the Entropy

I have a question regarding definition of entropy by expected value of the random variable $\log \frac{1}{p(X)}$:

$H(X) = E \log \frac{1}{p(X)}$,

where $X$ is drawn accordingly to the probability mass function of $p(x)$.

The problem is I am still don't understand two thing:

1) How this formula was derived from the original formula for the entropy

$H(X) = - \sum_{x \in X} p(x) \log p(x)$.

2) Even without knowing how to derive the second formula, what it the meaning of $p(X)$? Can you show how to find an entropy for the fair dice with one toss by the second formula.

-

1) Suppose $X$ is a random variable that takes only a finite number of values, say: $x_1$ with probability $p_1$, $x_2$ with probability $p_2, \ldots, x_n$ with probability $p_n$.

What is the expectation of $X$? Well, easy, $E(X) = \sum_{i=1}^n x_i p_i.$ What about the random variable $X^2$? Well, $E(X^2) = \sum_{i=1}^n x_i^2 p_i$.

The same goes for the random variable $p(X)$. You can think of it as taking the value $p_i$ whenever $X$ takes the value $x_i$. Therefore its expectation is $E(p(X)) = \sum_{i=1}^n p_i^2$.

Finally, the random variable $\log\left(\frac{1}{p(X)}\right) = - \log(p(X))$ has expectation $$E\left[\log\left(\frac{1}{p(X)}\right)\right] = E[-\log(p(X))] = \sum_{i=1}^n (- \log(p_i))p_i = -\sum_{i=1}^n p_i \log(p_i)$$

2) For instance, for the fair dice, $p(1) = p(2) = \ldots = p(6) = 1/6$, hence $$H(X) = - 6 \cdot \frac{1}{6}\log\left(\frac{1}{6}\right) = \log(6)$$

-
I have edited your post to replace $E(X)$ with $E(p(X))$ but this edit is too short to be accepted. –  Stéphane Laurent Mar 16 at 10:31
Done. Thank you. –  Pedro Milet Mar 16 at 13:38
{"extraction_info": {"found_math": true, "script_math_tex": 0, "script_math_asciimath": 0, "math_annotations": 0, "math_alttext": 0, "mathml": 0, "mathjax_tag": 0, "mathjax_inline_tex": 1, "mathjax_display_tex": 1, "mathjax_asciimath": 0, "img_math": 0, "codecogs_latex": 0, "wp_latex": 0, "mimetex.cgi": 0, "/images/math/codecogs": 0, "mathtex.cgi": 0, "katex": 0, "math-container": 0, "wp-katex-eq": 0, "align": 0, "equation": 0, "x-ck12": 0, "texerror": 0, "math_score": 0.9709537029266357, "perplexity": 112.47436188472183}, "config": {"markdown_headings": true, "markdown_code": true, "boilerplate_config": {"ratio_threshold": 0.18, "absolute_threshold": 10, "end_threshold": 15, "enable": true}, "remove_buttons": true, "remove_image_figures": true, "remove_link_clusters": true, "table_config": {"min_rows": 2, "min_cols": 3, "format": "plain"}, "remove_chinese": true, "remove_edit_buttons": true, "extract_latex": true}, "warc_path": "s3://commoncrawl/crawl-data/CC-MAIN-2013-48/segments/1386163050122/warc/CC-MAIN-20131204131730-00087-ip-10-33-133-15.ec2.internal.warc.gz"}