Doubt about Shannon entropy Let's consider a random variable X that can produce the output $c$ with probability $1/4$. I was told that $-\log_{2}(\frac{1}{4})$ represents the number of bits needed to represent $c$. But, imagine that X can produce the output $c$ with probability $1/4$ and the output $b$ with probability $3/4$, then I only need $1$ bit to express $c$, for example, $0=c$ (and $1=b$). Why am I wrong?
Without proof or even a precise definition: when a random variable has independent outputs with $p$ the probability to output $c$, then an optimal encoding of the source's outputs (one that minimizes the expected number of bits consumed to encode all successive values of a large number of outputs, including order) will use on average $-\log_2(p)$ bit to encode the output $c$. For example, $-\log_2(\frac14)=2$ bit for $p=\frac14$, as told to the OP.

As noted in the question, it is possible to make a different encoding that only uses $1$ bit to encode $c$, and that's less than $-\log_2(p)$ when $p&lt;\frac12$. But that different encoding will use more bits on average than an optimal encoding does, thus that observation does not contradict the statement in the previous paragraph.

For example, if the source outputs $c$ with probability $p=\frac14$ and $b$ with $p'=1-p=\frac34$, the encoding that outputs 0 for $c$ and 1 for $b$ requires $1$ bit per output, always, thus also on average. But it is possible to make a different encoding that consume less bits on average, down to the Shannon entropy of the source, which for two outputs is $-p\log_2(p)-(1-p)\log_2(1-p)\approx 0.81$ (see binary entropy function). An example of such encoding is to group outputs two by two, and encode as follows:
$$
\begin{array}{cc|l|cc}
\text{first} &amp; \text{second} &amp; \text{encoding} &amp; \text{size} &amp; \text{probability}\\
\text{output} &amp; \text{output}&amp; &amp; s &amp; q\\
\hline
b &amp; b &amp; \mathtt{0}   &amp; 1 &amp; \frac9{16}\\
b &amp; c &amp; \mathtt{10}  &amp; 2 &amp; \frac3{16}\\
c &amp; b &amp; \mathtt{110} &amp; 3 &amp; \frac3{16}\\
c &amp; c &amp; \mathtt{111} &amp; 3 &amp; \frac1{16}
\end{array}
$$

The probabilities $q$ sum to $1$, and it is possible to unambiguously decode a stream of bits output by this encoder.

The average number of bits for encoding two outputs is the sum of the products $s\,q$, that is $\frac{27}{16}$. The average number of bit per output is half that, that is $\frac{27}{32}\approx0.85$, which is less than $1$.



It's possible to prove that we can make encoders that arbitrarily approach the output bit per input symbol ratio of the Shannon entropy, but not encoders below that limit. It's also possible to rigorously define the average number of bits needed to represent an output value for a given encoder, then to show that this quantity must converge to $-\log_2(p_i)$ for encoders that approach optimality. That's involved, but we can at least justify that value.

Whatever sensible definition we give to the average number of bits $r_i$ needed to represent an output value $i$ for a given encoder, it must be such that we can compute the average number of bits produced by the encoder to represent an output of the source as $\sum p_i r_i$. We see that with $r_i=-\log_2(p_i)$, we get the standard formula for Shannon entropy:

$$S=-\sum p_i\log_2(p_i)$$

which justifies that in this formula, $-\log_2(p_i)$ is the average number of bits needed to represent an output value $i$. Which matches what the OP was explained.