Abstract
Abstract-The binomial and the Poisson distributions are shown to be maximum entropy distributions of suitably defined sets. Poisson's law is considered as a case of entropy maximization, and also convergence in information divergence is established.
I. INTRODUCTION
We shall use 5() to denote the Poisson distribution with mean , and b(n; p) to denote the binomial distribution with parameters (n; p).
We will not distinguish between a random variable and its distribution in the notation. Let X 1 ; X 2 ; . . . ; X n be a sequence of independent Bernoulli random variables, i.e., random variables with range f0; 1g. Define the success probabilities by p i = P (X i = 1), = n 1 p i , p max = max i p i , and S n = n 1 X i . We call S n an n-generalized binomial distribution and denote by Bn() the set of n-generalized binomial distributions with mean . Define the set of generalized binomial distributions B 1 () as the union B n () of all n-generalized binomial distributions.
Let P and Q be probability measures on f0; 1; 2; . . .g with point probabilities pi and qi, i = 0; 1; 2; . . . The basic properties of the information divergence are described, for instance, in [1] .
The convergence of the point probabilities of b(n; n ) to the point probabilities of 5() was established by Poisson. Convergence in total variation was studied by Prohorov [2] for the binomial distribution. Convergence of more general distributions are studied in [3] - [6] . See Steele [7] for a survey on the subject and further references. Information divergence does not define a metric but is related to total variation via Pinsker's inequality 1 2 kP 0 Qk 2 D(P kQ) proved by Csiszár [8] and others. If (Q n ) n2 is a sequence of probability distributions, we say that (Qn)n2 converges to Q in information divergence if D(Q n kQ) ! 0 for n ! 1. In Section II, it is shown that b(n; n ) converges to 5() in information divergence, and the proof is at least as simple as the proof of convergence in total variation. Pinsker's inequality shows that convergence in information divergence is a stronger condition than convergence in total variation. The use of information divergence also fits better together with the idea of maximum-likelihood estimation known from statistics.
The entropy of P is defined by
If is a set of distributions we define H() = sup P 2 (H(P)). In Section III, it is shown that both the binomial distributions and the Poisson distributions are maximum-entropy distributions on sets of generalized binomial distributions. Also Poisson's law is shown to be closely related to the maximum-entropy principle in the sense that Poisson's law can be formulated as "the entropy increases to its maximum." In this sense, these results are closely related to results about the central limit theorem obtained by Barron in [9] and Takano in [10] . . Therefore, D(Xk5())
II. POISSON'S LAW
is minimal for = E(X). Equivalently, = E(X) is the maximum-likelihood estimate given an empirical distribution according to X. Now it is convenient to define
If total variation is used to measure the difference between the distributions, the maximum-likelihood estimate is not the nearest distribution. In [11] - [13] , bounds on the total variation between the distribution of X and the nearest Poisson distribution are given.
Lemma 1:
For independent random variables X 1 and X 2 we have
Proof: First we observe that
where 5(1) and 5(2) are considered as independent Poisson distributions. The inequality (1) is obtained by data reduction of the map (X 1 ; X 2 ) ! X 1 + X 2 . Theorem 2: Let X 1 , X 2 1 11X n be a sequence of independent Bernoulli random variables. Define pi = P (Xi = 1), = n pi, and Sn = n Xi. Then 
III. MAXIMUM-ENTROPY DISTRIBUTIONS
In order to study the entropy of generalized binomial distributions, we need the following lemma which is a strengthening of a result obtained by Shepp 
A proof of (2) The last term is negative by concavity of the entropy function. We shall show that also the first term d du H(S n ) is less than or equal to 0.
Define b l = P (X3 + 1 11 + Xn = l). The lemma gives more evidence to the following conjecture stated by Shepp and Olkin [14, p. 4 ].
Conjecture 5:
The entropy H(S n ) is a concave function of the vector (p1; p2; . . . ; pn). 
Theorem 7:
The binomial distribution b(n; n ) is the maximum-entropy distribution in B n (), and for fixed H(b(n; n )) is increasing and H b n; n % H (B1()) ; for n ! 1:
Proof: There exists an n-generalized binomial distribution R with success probabilities ri where H(R) = H(Bn()). By Lemma 4 and symmetry we have that r i = r j for all i; j. Therefore, R is a binomial distribution with parameters (n; n ). To see that the sequence (H(b(n; n )))n2 is increasing we note that the sequence of sets (B n ()) n2 is increasing.
Proof: The geometric distribution is the maximum-entropy distribution among distributions with mean (see [16] ), so the Poisson distribution has entropy less than the entropy of the geometric distribution, which is finite. We have to show that H b n; n ! H (5()) ; for n ! 1:
We know that D b n; n 5() ! 0; for n ! 1 which implies b(n; n ; j) ! 5(; j) for n ! 1 for all j. which is an integrable upper bounding function with respect to the counting measure.
Remark 9: None of the sets Bn(), n = 3; 4; 5; . . . ; 1 are convex. If the sets B n () had been convex, we could have used Theorem 7 together with general results on entropy maximization obtained by Topsøe and others [16] - [18] to conclude that b(n; n ) converges to a distribution in cl(B 1 ()) in information divergence, without use of the results in Section II.
ACKNOWLEDGMENT
The author wants to thank Flemming Topsøe, Department of Mathematics, University of Copenhagen, for useful discussions.

{'paper_id': '16171405', 'title': 'Binomial and Poisson distributions as maximum entropy distributions', 's2_url': 'https://api.semanticscholar.org/CorpusID:16171405', 'source': 'S2ORC'}