Is it absurd to estimate entropy from a sample? To tell the exact entropy of a source, all I need to do is to use Shannon's formula $\sum -p(i) \lg p(i)$, where $i$ is the $i$-th element of the alphabet emitted by the source.  Therefore, the only thing that keeps me from telling the exact entropy is not knowing $p$.  So, the problem of estimating entropy reduces to the problem of estimating $p$.
I have studied Reid's answer to this question.  Reid seems to say that you get the sample 1011 you could have 0 to 4 bits of entropy.  Why would it be absurd to estimate the probability distribution from this sample?  It turns out we get three ones and one zero.  Is it absurd to guess that $p(1) = 3/4$ and $p(0) = 1/4$, therefore an estimate for the entropy of the source is $0.8111 = 1/4 \times (-\lg(1/4)) + (3/4 \times (-\lg(3/4)))$, and the amount of information in the sample is $3.244$ bits.
Theoretically, you can break down the question of estimating the entropy of a given collection of (assumed to be independent and identically distributed) samples into two steps:

Estimating the distribution of the underlying random variable

Computing that random variable's entropy


Generally you can do the first by &quot;counting&quot;. If you see the collection of 4 samples $0, 0, 0, 1$, you can set $\Pr[X = 0] = 3/4$, and $\Pr[X = 1] = 1/4$ (this is generally known as the &quot;empirical distribution&quot;). You can then easily compute the entropy.
Note that the rest of the question has a large caveat, in that you need a source of independent and identically distributed samples to apply it.
If you see $1011$, is this a single sample, or four independent, identically distributed samples?
To answer this you need to think carefully about how the samples are generated, but regardless I'll continue with discussing things assuming you can generate i.i.d. samples.
How accurate the entropy computation is therefore reduces to how close the empirical distribution is to the &quot;true&quot; underlying distribution.
For &quot;large enough&quot; sample sizes, it will converge to the true distribution, but quantifying the rate of convergence becomes important.
There are various ways to do this, a few are summarized in the empirical distribution function wikipedia page.
One particularly useful way to quantify this is via the DKW inequality.
Let $\mathcal{X}$ be the underlying (unknown) distribution, and let $X_1,\dots, X_n$ be $n$ i.i.d. samples from $\mathcal{X}$.
Let $F(x)$ be the cumulative distribution function of $\mathcal{X}$.
We define the empirical cumulative distribution function of the samples $X_1,\dots, X_n$ via:
$$F_n(x) = \frac{1}{n}\sum_{i = 1}^n \mathbf{1}_{X_i \leq x}$$
Here $\mathbf{1}_{X_i \leq x}$ is an &quot;indicator function&quot;, which is 1 if $X_i \leq x$, and 0 otherwise.
So $F_n(x)$ counts how many of the $X_i$ are less than $x$ (and then normalizes it to be in $[0,1]$ by dividing by $n$).
The DKW inequality then states that for any $\epsilon &gt; \sqrt{\frac{\ln(2)}{2n}}$:
$$\Pr[|\sup_{x\in \mathbb{R}} (F(x) - F_n(x))| &gt; \epsilon] \leq 2\exp(-2n\epsilon^2)$$
This gives a &quot;Chernoff-like&quot; bound on how far the cumulative distribution function can be from the empirical cumulative distribution function.
After estimating the empirical cumulative distribution function, you can convert this into estimates for the various probabilities.
This is because $p_i = \Pr[X = i] = \Pr[X \leq i] - \Pr[X \leq i-1] = F(i) - F(i-1)\approx F_n(i) - F_n(i-1) \pm 2\epsilon = \tilde{p}_i \pm 2\epsilon$.
More formally, by applying the DKW inequality we will get that $|p_i - \tilde{p}_i| \leq 2\epsilon$ with probability all but $2\exp(2n\epsilon^2)$.
We can then compute the entropy of this:
\begin{align*}
\mathbb{H}[\tilde{X}] &amp;= \sum_{i\in\mathsf{supp}(\tilde{X})} \tilde{p}_i(-\log_2(\tilde{p_i}))\\
&amp;= \sum_{i\in\mathsf{supp}(\tilde{X})} (p_i\pm 2\epsilon)(-\log_2(p_i\pm 2\epsilon))
\end{align*}
From here you could try to bound how close this is to the true entropy.
Unfortunately the only ways I currently see to do it are rather handwavy --- $-\log_2(x)$ is convex so $-\log_2(2(x+y)/2) \leq -1 -\log_2(x)/2 - \log_2(y)/2$, but $\pm\epsilon$ may be negative, so you start running into issues along those lines.
Anyway, you can proceed as you mention, but to get an accurate estimation of the entropy:

You need to be able to &quot;break&quot; your random source into independent and identically distributed samples
You need a large sample size (so the probability an estimate falls outside of the DKW inequality, $2\exp(-2n\epsilon^2)$, is &quot;small&quot;).