--- abstract: 'Commonly observed patterns typically follow a few distinct families of probability distributions. Over one hundred years ago, Karl Pearson provided a systematic derivation and classification of the common continuous distributions. His approach was phenomenological: a differential equation that generated common distributions without any underlying conceptual basis for why common distributions have particular forms and what explains the familial relations. Pearson's system and its descendants remain the most popular systematic classification of probability distributions. Here, we unify the disparate forms of common distributions into a single system based on two meaningful and justifiable propositions. First, distributions follow maximum entropy subject to constraints, where maximum entropy is equivalent to minimum information. Second, different problems associate magnitude to information in different ways, an association we describe in terms of the relation between information invariance and measurement scale. Our framework relates the different continuous probability distributions through the variations in measurement scale that change each family of maximum entropy distributions into a distinct family.' author: - 'Steven A. Frank' - Eric Smith bibliography: - 'entropy.bib' title: A simple derivation and classification of common probability distributions based on information symmetry and measurement scale --- Introduction ============ Commonly observed patterns follow a few families of probability distributions. For example, Gaussian patterns often arise from measures of height or weight, and gamma patterns often arise from measures of waiting times. These common patterns lead to two questions. How are the different families of distributions related? Why are there so few families, when the possible patterns are essentially infinite? These questions are important, because one can hardly begin to study nature without some sense of the fundamental contours of pattern and why those contours arise. For example, no one observing a Gaussian distribution of weights in a population would feel a need to give a special explanation for that pattern. The central limit theorem tells us that a Gaussian distribution is a natural and widely expected pattern that arises from measuring aggregates in a certain way. With other common patterns, such as power laws, the current standard of interpretation is much more variable. That variability arises because we do not have a comprehensive theory of how measurement and information shape the commonly observed patterns. Without a clear notion of what is expected in different situations, common and relatively uninformative patterns frequently motivate unnecessarily complex explanations, and surprising and informative patterns may be overlooked @frank09the-common. Currently, the differences between families of common probability distributions often seem arbitrary. Thus, little understanding exists with regard to how changes in process or in methods of observation may cause observed pattern to change from one common form into another. We argue that measurement, described by the relation between magnitude and information, unifies the distinct families of common probability distributions. Variations in measurement scale may, for example, arise from varying precision in observations at different magnitudes or from the way that information is lost when measurements are made on aggregates. Our unified explanation of the different commonly observed distributions in terms of measurement points the way to a deeper understanding of the relations between pattern and process. We develop the role of measurement through maximum entropy expressions for probability distributions. We first note that all probability distributions can be expressed by maximization of entropy subject to constraint. Maximization of entropy is equivalent to minimizing total information while retaining all the particular information known to constrain underlying pattern @jaynes57information [@jaynes57informationII; @jaynes03probability]. To obtain a probability distribution of a given form, one simply chooses the informational constraints such that maximization of entropy yields the desired distribution. However, constraints chosen to match a particular distribution only describe the sufficient information for that distribution. To obtain deeper insight into the causes of particular distributions and each distribution's position among related families of distributions, we derive the related forms of constraints through variations in measurement scale. Measurement scale expresses information through the invariant transformations of measurements that leave the form of the associated probability distribution unchanged @frank10measurement. Each problem has a characteristic form of information invariance and symmetry that sets the measurement scale @hand04measurement [@luce08measurement; @narens08meaningfulness] and the most likely probability distribution associated with that particular scale @frank10measurement. We show that measurement scales and the symmetries of information invariances form a natural hierarchy that generates the common families of probability distributions. We use *invariance* and *symmetry* interchangeably, in the sense that symmetry arises when an invariant transformation leaves an object unchanged @weyl52symmetry. The measurement hierarchy arises from two processes. First, we express the forms of information invariance and measurement scale through a continuous group of transformations, showing the relations between different types of information invariance. Second, the types of aggregation and measurement that minimize information and maximize entropy fall into two classes, each class setting a different basis for information invariance and measurement scale. The two types of aggregation correspond to the two major families of stable distributions that generalize the process leading to the central limit theorem: the <span>L<span>é</span>vy</span> family that includes the Gaussian distribution as a special case, and the Fisher-Tippett family of extreme value distributions. By expressing measurement scale in a general way, we obtain a wider interpretation of the families of stable distributions and a broader classification of the common distributions. Our derivation of probability distributions and their familial relations supersedes the Pearson and similar classifications of continuous distributions @johnson94continuous. Our system derives from a natural description of varying information in measurements under different conditions @frank10measurement, whereas the Pearson and related systems derive from phenomenological descriptions that generate distributions without clear grounding in fundamental principles such as measurement and information. Some recent systems of probability distributions, such as the unification by Morris @morris09unifying [@morris82natural], provide great insight into the relations between families of distributions. However, Morris's system and other common classifications do not derive from what we regard as fundamental principles, instead arising from descriptions of structural similarities among distributions. We provide a detailed analysis of Morris's system in relation to ours in Appendix C. We favor our system because it derives the relations between distributions from fundamental principles, such as maximum entropy and the invariances that define measurement scale. Although the notion of what is fundamental will certainly attract controversy, our favored principles of entropy, symmetries defined by invariances, and measurement scale certainly deserve consideration. Our purpose is to show what one can accomplish by starting solely with these principles. Maximum entropy and measurement scale ===================================== This section reviews our prior work on the roles of information invariance and measurement scale in setting observed pattern @frank10measurement. The following sections extend this prior work by expressing measurement in terms of the scale of aggregation and the continuous group transformations of information invariance. Maximum entropy --------------- The method of maximum entropy defines the most likely probability distribution as the distribution that maximizes a measure of entropy (randomness) subject to various information constraints @jaynes03probability. We write the quantity to be maximized as $$\label{eq:maxEnt} {\Phi}= {{\cal{E}}}- {\kappa}C_0 - \sum_{i=1}^n{\lambda}_iC_i,$$ where ${{\cal{E}}}$ measures entropy, the $C_i$ are the constraints to be satisfied, and ${\kappa}$ and the ${\lambda}_i$ are the Lagrange multipliers to be found by satisfying the constraints. Let $C_0=\int p_y{{\hbox{\rm d}}}y -1$ be the constraint that the probabilities must total one, where $p_y$ is the probability distribution function of $y$. The other constraints are usually written as $C_i= \int p_yf_i(y){{\hbox{\rm d}}}y -\bar{f}_i$, where the $f_i(y)$ are various transformed measurements of $y$, and the overbar denotes mean value. A mean value is either the average of some function applied to each of a sample of observed values, or an a priori assumption about the average value of some function with respect to a candidate set of probability laws. If $f_i(y)=y^i$, then $\bar{f}_i$ are the moments of the distribution—either the moments estimated from observations or a priori values of the moments set by assumption. The moments are often regarded as "standard" constraints, although from a mathematical point of view, any properly formed constraint can be used. Here, we confine ourselves to a single constraint of measurement. We express that constraint with a more general notation, $C_1= \int p_y{T}(f_y){{\hbox{\rm d}}}y -\bar{{T}}_f$, where $f_y\equiv f(y)$, and ${T}(f_y)\equiv{{T}_f}$ is a transformation of $f_y$. We could, of course, express the constraining function for $y$ directly through $f_y$. However, we wish to distinguish between an initial function $f_y$ that can be regarded as a standard measurement, in any sense in which one chooses to interpret the meaning of standard, and a transformation of standard measurements denoted by ${{T}_f}$ that arises from information about the measurement scale. The maximum entropy distribution is obtained by solving the set of equations $$\label{eq:maxEntSoln} {{{{{\partial}{\Phi}}\over{{\partial}p_y}}}} = {{{{{\partial}{{\cal{E}}}}\over{{\partial}p_y}}}} - {\kappa}- {\lambda}{{T}_f}=0,$$ where one checks the candidate solution for a maximum and obtains ${\kappa}$ and ${\lambda}$ by satisfying the constraint on total probability and the constraint on $\bar{{T}}_f$. We assume that we can treat the entropy measures and the maximization procedure by the continuous limit of the discrete case. In the standard approach, we define entropy by extension of Shannon information $$\label{eq:shannonDef} {{\cal{E}}}=-\int p_y\log\left({{{p_y}\over{m_y}}}\right){{\hbox{\rm d}}}y,$$ in which this expression may be called Jaynes's differential entropy @jaynes03probability, which is equivalent in form to the continuous expression of relative entropy or the Kullback-Leibler divergence @cover06elements. Here, we will interpret $m_y$ by information invariance and measurement scale as discussed below. With these definitions, the solution of <span>Eq. (\[eq:maxEntSoln\])</span> is $$\label{eq:shannonSoln} p_y \propto m_ye^{- {\lambda}{{T}_f}},$$ where ${\lambda}$ satisfies the constraint $C_1$, and the proportionality is adjusted so that the total probability is one by choosing the parameter ${\kappa}$ to satisfy the constraint $C_0$. Information invariance and measurement scale {#sectionInvar} -------------------------------------------- Maximum entropy must capture all of the available information about a particular problem. One form of information concerns transformations to the measurement scale that leave the most likely probability distribution unchanged @jaynes03probability [@frank09the-common; @frank10measurement]. Here, it is important to distinguish between measurements and measurement scale. In our notation, we start with measurements, $f_y$, made on the measurement scale $y$. For example, we may have measures of squared deviations about zero, $f_y=y^2$, with respect to the measurement scale $y$, such that $\bar{f}_y$ is the second moment of the measurements with respect to the underlying measurement scale. Suppose that we obtain the same information about the underlying probability distribution from measurements of $f_y$ or transformed measurements, ${G}(f_y)$. Put another way, if one has access only to measurements ${G}(f_y)$, one has the same information that would be obtained if the measurements were reported as $f_y$. We say that the measurements $f_y$ and ${G}(f_y)$ are equivalent with respect to information, or that the transformation $f_y \rightarrow {G}(f_y)$ is an information invariance that describes a symmetry of the measurement scale. To capture this information invariance in maximum entropy, we must express our measurements so that $$\label{eq:transDef} {T}(f_y) = {\delta}+ {\phi}{T}[{G}(f_y)]$$ for some arbitrary constants ${\delta}$ and ${\phi}$ @frank10measurement. Putting this definition of ${T}(f_y)\equiv{{T}_f}$ into <span>Eq. (\[eq:shannonSoln\])</span> shows that we get the same maximum entropy solution whether we use the observations $f_y$ or the transformed observations, ${G}(f_y)$, because the ${\kappa}$ and ${\lambda}$ constants will adjust to the constants ${\delta}$ and ${\phi}$ so that the distribution remains unchanged. Deriving probability distributions ================================== The prior section established two key steps. First, maximum entropy probability distributions have the form given in <span>Eq. (\[eq:shannonSoln\])</span> as $p_y \propto m_ye^{- {\lambda}{{T}_f}}$. Second, the expression of ${T}(f_y)$ for each problem comes from the particular information invariance ${G}(f_y)$ associated with that particular problem. To derive specific probability distributions, we must pass three further steps, which we develop in the following sections. First, we turn the abstract notions of information invariance and measurement scale into specific expressions for the measurement scale function, ${T}(f_y)$. We accomplish this by developing the continuous group transformations for information invariance. Those continuous transformations provide an abstract hierarchy of forms for probability distributions based on the scale factor, $m_y$, the specific measured attribute, $f_y$, and how the information and precision of measurements change with magnitude expressed by the measurement scale ${T}(f_y)$. Second, we define $m_y$ as the relation between the scale of information invariance and the scale on which we express probability. To use the maximization of entropy and the associated minimization of information, we must relate the information invariance of measurement to the scale on which underlying processes dissipate information. We consider alternative interpretations of scale that may be associated with the dissipation of information by aggregation of random perturbations and by measurements of extreme values. We also consider measurements on a scale that differs from the basis for dissipation of information. Third, we consider how to interpret $f_y$, which is the value used to describe the informational constraint in relation to the measurement scale ${T}(f_y)$, leading to the constraint $\bar{{T}}_f$. We discuss $f_y$ as a reduction in the dimensionality of information to a single sufficient dimension. That sufficient dimension sets the form of probability under the various processes of information dissipation that lead to the common probability distributions. Continuous group transformations of measurement =============================================== The transformation in <span>Eq. (\[eq:transDef\])</span> sets the relation between information invariance and measurement scale. However, that expression does not show in a simple way the relations between information and measurement. To understand commonly observed patterns in relation to the families of probability distributions, it is helpful to express in a general way the underlying symmetry that determines information invariance and measurement scale. From that underlying symmetry, we may see more clearly the associated relations between the forms of probability distributions. The affine structure -------------------- The relation between information invariance and measurement scale in <span>Eq. (\[eq:transDef\])</span> arises directly from the form of maximum entropy solutions in <span>Eq. (\[eq:shannonSoln\])</span>, in which probability distributions are exponentials of the transformed constraint measures, ${T}_f$. In particular, the probability distribution associated with a constraint is invariant to an additive shift of the constraint and a multiplicative change in the scale of the constraint, given by the parameters ${\delta}$ and ${\phi}$ in <span>Eq. (\[eq:transDef\])</span>. It is that symmetry in the affine structure of invariant transformation that ultimately sets the underlying relations between information, measurement, and the familial forms of the common probability distributions. To understand the affine structure of the invariant transformation in <span>Eq. (\[eq:transDef\])</span> more clearly, we can express that invariant transformation as a continuous operator. First, rearrange <span>Eq. (\[eq:transDef\])</span> as an equivalent expression $$\label{eq:transDefInv} {T}[{G}(f_y)] = a + b{T}(f_y)$$ with new parameters $a$ and $b$ that are easily calculated from <span>Eq. (\[eq:transDef\])</span>. We show in Appendix A that we can express the same information invariance of ${G}(f_y)$ by the differential operator defined as $$v_w = \left( \alpha + \beta {T}\right) \frac{{{\hbox{\rm d}}}}{{{\hbox{\rm d}}}{T}} \label{eq:v_alpha_beta_form}$$ that can be applied to ${T}$ as $$v_w \! \left( {T}\right) = \alpha + \beta{T}. \label{eq:v_alpha_beta_onT}$$ Recursive application of $v_w$ preserves the affine structure and so keeps the successive transformations within the group of admissible invariance relations. We can express $v_w$ as $$v_w = \frac{{{\hbox{\rm d}}}}{{{\hbox{\rm d}}}w} , \label{eq:v_w_form}$$ where $w\equiv w (f_y)$ is some function of $f_y$. We then have a differential equation for ${T}$ as $$\frac{{{\hbox{\rm d}}}{T}}{{{\hbox{\rm d}}}w} - \beta {T}= \alpha , \label{eq:T_resp_genform}$$ which has solutions of the general form $${T}\! \left( f_y\right) = {T}_0 e^{\beta w} + \frac{\alpha}{\beta} \left( e^{\beta w} - 1 \right) , \label{eq:T_gen_soln}$$ which as $\beta \rightarrow 0$ goes to ${T}(f_y) \rightarrow {T}_0 + \alpha w$. <span>Eq. (\[eq:T\_gen\_soln\])</span> gives the most general class of measurement functions, ${T}(f_y)$, for which the associated transformations generated by $v_w$ preserve information invariance. The operator $v_w$ can be applied repeatedly, creating a recursively generated sequence of deformations that all satisfy the fundamental relation between deformations of measurement and information invariance. By thinking of $w(f_y)$ as a parameter that expresses the deformation of measurement associated with a measurement scale, ${T}(f_y)$, we can create a sequence in which each successive deformation corresponds to a successive class of probability distributions with familial relations to each other defined by the structure of the sequence of deformations to $w(f_y)$. The general form of probability distributions --------------------------------------------- From <span>Eq. (\[eq:shannonSoln\])</span>, the maximum entropy solution is $$\label{eq:shannonSoln2} p_y \propto m_ye^{-{\lambda}{{T}_f}}.$$ From <span>Eq. (\[eq:T\_gen\_soln\])</span>, we can now express the maximum entropy solution as $$\label{eq:master} p_y \propto m_y e^{-{\Lambda}e^{{\beta}w}},$$ where ${\Lambda}= {\lambda}({T}_0+\alpha/{\beta})$, and $w\equiv w(f_y)$. In the limit ${\beta}\rightarrow 0$, this becomes $$p_y \propto m_y e^{-{\gamma}w}$$ where ${\gamma}={\lambda}{\alpha}$. In Appendix B we describe the case of extreme values, for which we will use $m_y={{\hbox{\rm d}}}{T}(f_y)/{{\hbox{\rm d}}}y$. When $f_y=y$ and $m_y={{\hbox{\rm d}}}{T}(y)/{{\hbox{\rm d}}}y={T}'$, it will be convenient to write $$\label{eq:Tprime} {T}' \propto w'e^{{\beta}w},$$ where $w'={{\hbox{\rm d}}}w(y)/{{\hbox{\rm d}}}y$, and as ${\beta}\rightarrow0$, ${T}' \propto w'$. Intuitive description of measurement and probability ==================================================== Intuitively, one can think of the symmetry of information invariance and measurement scale in the following way. On a linear scale, each incremental change of fixed length yields the same amount of information or surprise independently of magnitude. Thus, if we change the scale by multiplying all magnitudes by a constant, we obtain the same pattern of information relative to magnitude. In other words, the linear scale is invariant to multiplication by a constant factor so that, within the framework of maximum entropy subject to constraint, we get the same information about probability distributions from an observation $y$ or ${G}(y)=cy$. In this section, we use $f_y=y$ to isolate the symmetry expressed by particular choices of ${T}$ and ${G}$. On a logarithmic scale, each incremental change in proportion to the current magnitude yields the same amount of information or surprise. Information is scale dependent. We obtain the same information at any point on the scale by comparing ratios. For example, we gain the same information from the increment ${{\hbox{\rm d}}}y/y={{\hbox{\rm d}}}\log(y)$ independently of the magnitude of $y$. Thus, we achieve information invariance with respect to ratios by measuring increments on a logarithmic scale. Within the framework of maximum entropy subject to constraint, we get the same information about probability distributions from an observation $y$ or ${G}(y)=y^c$, corresponding to informationally equivalent measurements ${T}(y)=\log(y)$ and ${T}(y^c)=c\log(y)$ (see ref. @frank10measurement). The form of a probability distribution under maximum entropy can be read directly as an expression of how the measurement scale changes with magnitude. From the general solution in <span>Eq. (\[eq:shannonSoln\])</span>, linear scales ${T}(y)\propto y$ yield distributions that are exponential in $y$, whereas logarithmic scales ${T}(y)\propto c\log(y)$ yield distributions that are linear in $y^c$. Exponential distributions of the form $e^{-{\lambda}y}$ arise from underlying linear scales, whereas power law distributions of the form $y^{-c}$ arise from underlying logarithmic scales. Many common distributions have compound form, in which one can read directly how the underlying measurement scale changes with magnitude. For example, the gamma distribution has form $y^{-c}e^{-{\lambda}y}$. When the magnitude of $y$ is small, the shape of the distribution is dominated by the power law component, $y^{-c}$. As the magnitude of $y$ increases, the shape of the distribution is dominated by the exponential component, $e^{-{\lambda}y}$. Thus, the underlying measurement scale grades from logarithmic at small magnitudes to linear at large magnitudes. Indeed, the gamma distribution is exactly the expression of an underlying measurement scale that grades from logarithmic to linear as magnitude increases. Nearly every common probability distribution can be read directly as a simple expression of the change in the underlying measurement scale with magnitude. Hierarchies of common probability distributions =============================================== Given a particular form for the function $w(f_y)$, the measurement scale ${T}(f_y)$ follows from <span>Eq. (\[eq:T\_gen\_soln\])</span> and the associated probability distribution follows from <span>Eq. (\[eq:master\])</span>. Although we can choose $w$ in any way that we wish, certain measurement scales and information invariances are likely to be common. We discussed in our earlier paper the importance two scales @frank10measurement. The first scale grades from linear to logarithmic as magnitude increases, which we call the linear-log scale. The second scale inverts the linear-log scale to be logarithmic at small magnitudes and linear at large magnitudes, giving the log-linear scale. The inversion relating the two scales can be expressed by a Laplace transform, showing the natural duality of the scales and a connection to recent studies on superstatistics @frank10measurement. The linear-log scale -------------------- In terms of the notation in the present paper, we can define $w$ to establish a hierarchy of measurement deformations, in which each level in the hierarchy arises from successive application of the linear-log scaling to the scale in the previous level in the hierarchy. To define the linear-log measurement function in terms of $w$, note from <span>Eq. (\[eq:T\_gen\_soln\])</span> that, as ${\beta}\rightarrow0$, the forms of $w$ and the measurement function ${T}$ become equivalent with respect to setting the associated probability distribution. Thus, by setting $w$, we are defining the limiting form of the measurement function. With these issues in mind, define $$w^{(i)}=\log\left(c_i+w^{(i-1)}\right),$$ with $w^{(0)} = f_y$. The constant $c_i$ sets the transition between linear and logarithmic scaling: the scale is linear when $w^{(i-1)}$ is small relative to $c_i$ and logarithmic when $w^{(i-1)}$ is large relative to $c_i$. As $c_i\rightarrow0$, we can use $w^{(i)}=\log\left(w^{(i-1)}\right)$. It is easiest to see the abstract structure of the measurement hierarchy and the associated forms of probability distributions in the limiting case $c_i\rightarrow0$, leading to purely logarithmic deformations. The first row of Table \[tab:pureLog\] begins with the base measurement $w^{(0)}=f_y$. The following two rows show the first two deformations for the sequence $i=0,1,2$. $w(f_y)$ $p_y$ $p_{y|\beta \rightarrow 0} $ ----------------- ----------------------------------------------------- ------------------------------------------ \[-12pt\] $f_y$ $ m_y e^{-{\Lambda}e^{{\beta}f_y}} $ $ m_y e^{-{\gamma}f_y} $ $\log f_y$ $ m_y e^{-{\Lambda}f_y^{\beta}} $ $ m_y f_y^{-{\gamma}} $ $\log\log f_y$ $ m_y e^{-{\Lambda}\left(\log f_y\right)^{\beta}} $ $ m_y\left(\log f_y\right)^{-{\gamma}} $ : The logarithmic measurement hierarchy and the associated form of the probability distribution function $p_y$ from <span>Eq. (\[eq:master\])</span>. Note that ${\beta}\rightarrow0$ of each line corresponds to ${\beta}=1$ of the following line. \[tab:pureLog\] This table gives the hierarchy of probability distributions that arise from successive logarithmic deformations. With this structure in mind, we give the full expansion with $c_i\ne0$ in Table \[tab:linearLog\]. \[ht\] $w(f_y)$ $p_y$ $p_{y|\beta \rightarrow 0} $ -------------------------------------------------- -------------------------------------------------------------------------- --------------------------------------------------------------- \[-12pt\] $f_y$ $ m_y e^{-{\Lambda}e^{{\beta}f_y}} $ $ m_y e^{-{\gamma}f_y} $ $\log \left(c_1+f_y\right)$ $ m_y e^{-{\Lambda}\left(c_1+f_y\right)^{\beta}} $ $ m_y\left(c_1+f_y\right)^{-{\gamma}} $ $\log\left(c_2+\log\left(c_1+ f_y\right)\right)$ $ m_y e^{-{\Lambda}\left(c_2+ \log\left(c_1+f_y\right)\right)^{\beta}} $ $ m_y\left(c_2+ \log\left(c_1+f_y\right)\right)^{-{\gamma}} $ We discuss the interpretation of $m_y$ and $f_y$ below. The different interpretations of those values lead directly to specific forms for probability distributions. Before interpreting $m_y$ and $f_y$, we present an alternative measurement scale. The log-linear scale -------------------- We obtain the log-linear measurement deformation hierarchy @frank10measurement from $$w^{(i)}=c_i w^{(i-1)}+\log\left(w^{(i-1)}\right),$$ from which we obtain the probability distributions in Table \[tab:logLinear\]. The log-linear scale changes logarithmically at small magnitudes and linearly at large magnitudes. \[ht\] $w(f_y)$ $p_y$ $p_{y|\beta \rightarrow 0} $ -------------------------------------------------------------------- ---------------------------------------------------- -------------------------------------------- \[-12pt\] $f_y$ $ m_y e^{-{\Lambda}e^{{\beta}f_y}} $ $ m_y e^{-{\gamma}f_y} $ $c_1 f_y+\log f_y$ $ m_y e^{-{\Lambda}f_y^{\beta}e^{c_1{\beta}f_y}} $ $ m_y f_y^{-{\gamma}}e^{-c_1{\gamma}f_y} $ $c_2\left(c_1f_y+\log f_y\right)+\log\left(c_1f_y+\log f_y\right)$ $ m_y e^{-{\Lambda}e^{{\beta}w}} $ $ m_y e^{-{\gamma}w}$ Other scales ------------ The linear-log and log-linear scales describe common forms of measurement functions. In this section, we briefly mention some other scales listed in Table \[tab:otherScales\]. These additional scales illustrate the ways in which measurement relates to the patterns of probability. The first line of Table \[tab:otherScales\] shows a log-linear-log scale for a measure on the interval $(c_1,c_2)$. That scale changes logarithmically near the boundaries and linearly near the middle of the range, in which $\log b$ describes the skew in the scaling pattern. The second line of Table \[tab:otherScales\] shows a linear-log-linear scale for $f_y > 0$. That scale changes linearly near the lower boundary of zero, linearly at large magnitudes, and logarithmically at intermediate values. \[ht\] $w(f_y)$ $p_y$ $p_{y|\beta \rightarrow 0} $ --------------------------------------------------- -------------------------------------------------------------- ----------------------------------------------------- \[-12pt\] $\log\left((c_2-f_y)(f_y-c_1)^b\right)$ $ m_y e^{-{\Lambda}(c_2-f_y)^{\beta}(f_y-c_1)^{b{\beta}}} $ $ m_y (c_2-f_y)^{-{\gamma}}(f_y-c_1)^{-b{\gamma}} $ $c_2f_y+b\log(c_1+f_y)$ $ m_y e^{-{\Lambda}(c_1+f_y)^{b{\beta}} e^{c_2{\beta}f_y}} $ $ m_y (c_1+f_y)^{-b{\gamma}} e^{-c_2{\gamma}f_y} $ The scale of information ======================== The prior section presented probability distributions in terms of $m_y$ and $f_y$. This section develops the interpretation of $m_y$, which arises from the relation between the scale of information invariance and the scale on which we express probability. The key issue is that maximum entropy requires some underlying process to dissipate information. With regard to deriving probability distributions, we may consider three aspects of scale in relation to the dissipation of information. First, we may measure an outcome that arises from the aggregation of a series of random perturbations. Second, we may measure only the extreme values of some underlying process, thereby throwing away all information about the underlying process except the form of the upper or lower tail of the underlying distribution. Third, the dissipation of information may occur on one scale, but we may wish to make our measurements with respect to another scale. Each of these three aspects of the scale of information dissipation leads to a simple interpretation of probability measure in maximum entropy analysis. We give a brief description each scale of information dissipation in relation to calculating $m_y$. Aggregation of perturbations ---------------------------- In the standard application of maximum entropy, accumulation of random perturbations without constraint leads to a uniform probability measure, which has maximum entropy and minimum information. Thus, the scale at which information dissipates is the same as the scale of the probability measure. In this case, our formulation of maximum entropy has $m_y\equiv1$, because any information that arises from deformation of measurement relative to the uniform default is included in our expression of measurement scale, ${T}(f_y)$. Extreme values -------------- The distribution of extreme values depends only on the total (integral) of the probability measure in the tail of an underlying probability distribution @embrechts97modeling. Because extreme value distributions arise from integrals of probability measures, the dissipation of information and the associated measurement scale for extreme values is expressed in terms of the cumulative distribution function (see Appendix B). To obtain the associated form of the probability measure with respect to the probability distribution function, $p_y$, we must transform the invariant measurement scale originally expressed with respect to the integral of the underlying probability measure. To change from the integral scale of the cumulative distribution to the scale of the probability measure associated with the probability density function, we simply differentiate the initial measurement scale, ${T}(f_y)$, from the cumulative distribution scale to obtain the associated change in probability measure (Appendix B). For $f_y=y$, we obtain $m_y = {{\hbox{\rm d}}}{T}(y)/{{\hbox{\rm d}}}y = {T}'$. We gave the general form of $m_y={T}'$ in <span>Eq. (\[eq:Tprime\])</span>. Change of variable ------------------ In some cases, information may dissipate on one scale, but we choose to express probability on another scale. The log-normal distribution is the classic example. Using Table \[tab:pureLog\], we may consider measurements that lead to the Normal or Gaussian distribution by either analyzing squared deviations from a central value, $f_y=(y-{\mu})^2$ in line one of Table \[tab:pureLog\] with ${\beta}\rightarrow0$ or, equivalently, linear perturbations of $f_y=(y-{\mu})$ in line two of Table \[tab:pureLog\] with ${\beta}=2$. In these cases, the perturbations are direct measures rather than the tail probabilities of extreme values, so $m_y=1$, and we have the standard form of the Gaussian as $p_y\propto e^{-{\gamma}(y-{\mu})^2}$. If we prefer to analyze values on a logarithmic scale, then we make the transformation $y\rightarrow\log y$. This case does not arise from invariant information and the associated measurement transformation, but rather from a change of variable to a different scale. So we must change our measure, as in any standard change of variable. In this case, the change of measure is $m_y{{\hbox{\rm d}}}y = {{\hbox{\rm d}}}\log y={{\hbox{\rm d}}}y/y$, thus $m_y=y^{-1}$ and we obtain the log-normal distribution $p_y \propto y^{-1}e^{-\tilde{{\gamma}}\left(\log y - \tilde{{\mu}} \right)^2}$, where $\tilde{{\gamma}}$ and $\tilde{{\mu}}$ are transformed appropriately. Sufficiency: reduction of information ===================================== The algorithm of maximum entropy allows us to choose any constraint ${T}(f_y)$. However, one of our main goals is to provide a clear rationale for the choice of constraint, so that maximum entropy is more than a tautological description of probability distributions. We have expressed the choice of the measurement scale, ${T}$, in terms of information invariance set by the underlying problem. Although information invariance may take various forms, we followed our earlier paper @frank10measurement in which we defended the linear-log and log-linear scales as likely to be common scales associated with common information invariances. Once we have set the transformation ${T}(f_y)$ by these common information invariances, many widely observed probability distributions follow. In some cases, deriving probability distributions requires using an observable, $f_y\ne y$, that differs from the scale $y$ of the underlying probability measure. For example, we may use the squared deviations from a central location, or a fractional moment $f_y=y^{\alpha}$, where ${\alpha}$ is not an integer @frank09the-common. Use of $f_y=y$ or of squared deviations $f_y=(y-{\mu})^2$ is widely accepted. Such choices lead to $f_y$ being a sufficient reduction of all of the information in observations in order to express common probability distributions. For our purposes in this paper, we simply note that we can derive many common distributions by the widely accepted use of $f_y=y$ or $f_y$ as a squared deviation. But the reasons that particular choices of $f_y$ are sufficient have not been fully explained with regard to maximum entropy, particularly fractional moments such as $f_y=y^{\alpha}$ @frank09the-common. Those reasons probably have to do with the sort of analysis described by large deviation theory @touchette09the-large, in which the retained information arises from the minimal descriptions of location and scale that remain when one normalizes the consequences of a sequence of perturbations so that one obtains a stable limiting form. \[\] Conclusions =========== Table \[tab:commonDistn\] shows many of the commonly observed probability distributions. Those distributions arise directly from maximum entropy applied to various natural measurement scales. The measurement scales express information invariances associated with particular types of problems and the scale on which information dissipation occurs. We confined ourselves to various combinations of linear and logarithmic scaling, which were sufficient to express many common distributions. Our method readily extends to other types of information invariance and measurement scale and their associated probability distributions. Acknowledgements {#acknowledgements .unnumbered} ================ SAF is supported by National Science Foundation grant EF-0822399, National Institute of General Medical Sciences MIDAS Program grant U01-GM-76499, and a grant from the James S. McDonnell Foundation. DES thanks Insight Venture Partners for support. Appendices {#appendices .unnumbered} ========== On the association between measurement functions and classes of scale transformations ===================================================================================== If the transformation $f_y \rightarrow G \! \left( f_y \right)$ is an invariance of a measurement function $T$, it is clear that repeated applications of $G$, expressed as $G \circ G, G \circ G \circ G,\ldots$, are also invariances of $T$. It is the larger group of invariances that we wish to identify with the measurement scale that defines $T$, and not only a single transformation. To simplify notation in this Appendix, we use $f_y=y$. The same analysis applies to $f_y$. In general, making a unique association between a transformation $G$ and a measurement function $T$ is inconvenient for finite transformations, because $G$ combines a magnitude and a direction of deformation. The magnitude is added under compositions $G \circ G \ldots$, while the direction remains invariant. As we will derive below, the relevant measure of the magnitude of a transformation as in <span>Eq. (\[eq:transDefInv\])</span> will be $\sim \log b$, and the relevant measure of direction will be $a / \left( b-1 \right)$. To isolate the direction of $G$ that may be associated with a measurement function $T$, we work with infinitesimal rather than finite affine transformations. Infinitesimal transformations are constructed from <span>Eq. (\[eq:transDefInv\])</span> in the text by writing $a \equiv \epsilon \alpha$, $\left( b-1 \right) \equiv \epsilon \beta$, and then taking $\epsilon \rightarrow 0$ for fixed $\alpha$ and $\beta$. An infinitesimal transformation $G^{\epsilon}$ then satisfies <span>Eq. (\[eq:transDefInv\])</span> in the form $$T \! \left[ G^{\epsilon} \! \left( y \right) \right] = T \! \left( y \right) + \epsilon \left[ \alpha + \beta T \! \left( y \right) \right] . \label{eq:T_G_inf_def}$$ $G$ itself must therefore also be infinitesimally different from the identity, and must have the form $$G^{\epsilon} \! \left( y \right) = y + \epsilon v \! \left( y \right) . \label{eq:G_inf_on_y}$$ for some function $v \! \left( y \right)$. We introduce a quantity $\hat{v}$ called the *generator* of the deformation, such that the operator $e^{\epsilon \hat{v}}$ generates the infinitesimal transformations Eqs. (\[eq:T\_G\_inf\_def\],\[eq:G\_inf\_on\_y\]), and such that finite transformations $G$ or affine transformations <span>Eq. (\[eq:transDefInv\])</span> are produced by the exponential operation of $\hat{v}$ with non-infinitesimal $\epsilon$. Compounding a function corresponds to addition of parameters $\epsilon$, as may be checked from the power-series definition of $e^{\epsilon \hat{v}}$ within its radius of convergence. We define a *representation* of the generator $\hat{v}$ as an explicit differential operator that produces the correct transformation on the argument $y$ or $T \! \left( y \right)$, as appropriate. The two representations of the generators are related as $$\begin{aligned} T \! \left[ y + \epsilon v \! \left( y \right) \right] & = & \left[ 1 + \epsilon v \! \left( y \right) \frac{{{\hbox{\rm d}}}}{{{\hbox{\rm d}}}y} \right] T \! \left( y \right) \nonumber \\ & = & \left[ 1 + \epsilon \left( \alpha + \beta T \right) \frac{{{\hbox{\rm d}}}}{{{\hbox{\rm d}}}T} \right] T . \nonumber \\ \label{eq:T_G_inf_reps}\end{aligned}$$ From the requirement that the two expressions produce the same result, we may assign the representations $$\begin{aligned} \hat{v} & \leftrightarrow & v \! \left( y \right) \frac{{{\hbox{\rm d}}}}{{{\hbox{\rm d}}}y} \equiv \frac{{{\hbox{\rm d}}}}{{{\hbox{\rm d}}}w} \nonumber \\ & \leftrightarrow & \left( \alpha + \beta T \right) \frac{{{\hbox{\rm d}}}}{{{\hbox{\rm d}}}T} \label{eq:v_reps}\end{aligned}$$ for some function $w \! \left( y \right) = \int^y {{\hbox{\rm d}}}y^{\prime} 1 / v \! \left( y^{\prime} \right)$. Regarding $T$ as a function of argument $w$ rather than $y$, and setting equal the two coefficients of $\epsilon$ in <span>Eq. (\[eq:T\_G\_inf\_reps\])</span>, we obtain a relation between any function $w \! \left( y \right)$, coefficients $\alpha$ and $\beta$, and the function $T$ in the form $$\frac{{{\hbox{\rm d}}}T}{{{\hbox{\rm d}}}w} = \alpha + \beta T . \label{eq:reps_w_to_diff_eq_T}$$ This is rearranged to produce <span>Eq. (\[eq:T\_resp\_genform\])</span>. From the solutions to <span>Eq. (\[eq:reps\_w\_to\_diff\_eq\_T\])</span>, we may readily check that the action of the transformation $e^{\epsilon \hat{v}}$ for arbitrary $\epsilon$ (not necessarily small) is $$T \! \left[ G \! \left( y \right) \right] = e^{\epsilon \hat{v}} T \! \left( y \right) = \frac{\alpha}{\beta} \left( e^{\epsilon \beta} - 1 \right) + e^{\epsilon \beta} T \! \left( y \right) , \label{eq:finite_trans_T}$$ from which we recover expressions for the coefficients $a$ and $b$ in <span>Eq. (\[eq:transDefInv\])</span>. Under composition $G \rightarrow G \circ G$, the parameter $\epsilon \rightarrow 2 \epsilon$. The composition rules for $a$ and $b$ under composition of $G$ may be worked out easily, but depending on the function $w \! \left( y \right)$, the direct composition of finite transformations $G$ on $y$ may be quite complicated. Information measures for cumulative distributions {#sec:cum_dist} ================================================= The presence of the measure $m_y$ in the probability density function in <span>Eq. (\[eq:shannonSoln2\])</span> complicates the discussion of measurement invariance, because in the general case $m_y$ is not required to obey any prescribed transformation when $f_y \rightarrow G \! \left( f_y \right)$. In general, $y$ need not even be a numerical index, whereas $T \! \left( f_y \right)$ is necessarily numerical because it is proportional to an information measure $- \log \left( p_y / m_y \right)$. The class of cases in which the measurement function, $T$, completely controls the properties of $p_y$ are those in which measurement constrains the *cumulative* probability distribution function rather than the probability density function. For these cases $m_y$ is not independent, but is given in terms of $T$ and $f_y$, as we now show. Relative entropy is ordinarily defined for the probability density. However, if we set $$m_y = \frac{ {{\hbox{\rm d}}}T \! \left( f_y \right) }{ {{\hbox{\rm d}}}y } = \frac{{{\hbox{\rm d}}}f_y}{{{\hbox{\rm d}}}y} T^{\prime} \! \left( f_y \right) , \label{eq:m_y_cum_ent}$$ then $m_y$ becomes a Lebesgue measure on $y$ with respect to the increment ${{\hbox{\rm d}}}T$. The probability density from <span>Eq. (\[eq:shannonSoln2\])</span> becomes $$p_y \propto \frac{{{\hbox{\rm d}}}}{{{\hbox{\rm d}}}y} e^{- \lambda T \left( f_y \right) } . \label{eq:dens_from_cum}$$ <span>Eq. (\[eq:dens\_from\_cum\])</span> defines the relation between a probability density and its cumulative distribution, meaning that under a suitable ordering of $y$, we may take $e^{- \lambda T\left( f_y \right) }$ to be the cumulative distribution. With this choice of measure, the relative entropy $\mathcal{E}$ from <span>Eq. (\[eq:shannonDef\])</span> becomes $$\begin{aligned} - \int {{\hbox{\rm d}}}y\, p_y \log \left( \frac{p_y}{m_y} \right) & = & - \int {{\hbox{\rm d}}}y \frac{{{\hbox{\rm d}}}T}{{{\hbox{\rm d}}}y} \left( \frac{p_y}{{{\hbox{\rm d}}}T / {{\hbox{\rm d}}}y} \right) \log \left( \frac{p_y}{{{\hbox{\rm d}}}T / {{\hbox{\rm d}}}y} \right) \nonumber \\ & = & - \int {{\hbox{\rm d}}}T p_T \log p_T , \label{eq:rel_ent_T}\end{aligned}$$ in which $p_T$ is the probability density defined on the variable $T$. Since the maximum-entropy solution is always exponential in $T$, the relative entropy of <span>Eq. (\[eq:rel\_ent\_T\])</span> is effectively an information function for the cumulative distribution. An application in which constraints under aggregation apply by construction to the cumulative distribution is the computation of extreme-value statistics @kotz00extreme. The cumulative probability distribution for the maximizer or minimizer of a sample of $n$ realizations of a random variable is the product of $n$ factors of the cumulative distribution for a single realization. It was also noted in ref. @frank09the-common that the relative entropy may be evaluated on the characteristic function (Fourier or Laplace transform) of a distribution, and that the maximum-entropy solutions in the transformed domain are the Lévy stable distributions. The characteristic function at frequency argument $k = 0$ always takes value unity. Therefore it, like a cumulative distribution, has a reference normalization of unity, and indeed, the symmetric Lévy-stable distributions @sato01basic correspond in form to the Weibull family of extreme value distributions. Both are obtained within our classification for $m_y$ defined by <span>Eq. (\[eq:m\_y\_cum\_ent\])</span>, for suitable reductions $f_y$. The Morris Natural Exponential Families in relation to entropy-maximizing distributions ======================================================================================= Symmetry-based approaches to select or to classify probability distributions ---------------------------------------------------------------------------- Many systems, since Pearson's, for either selecting or classifying probability distributions, have been based on symmetry groups, as our method is. (Pearson's system may be seen as one based on the analytic structure of the log-probability, a criterion that we will return to consider in a moment.) The systems differ in generality, depending on the space in which the symmetry group acts, and depending on whether it constrains a single distribution or a family. Two methods based on symmetry (ours and that of Carl Morris, described below) have interpretations in terms of *scale invariance* of observables. Both systems collect probability distributions into families, whose members differ only by a scale factor. A third approach (known as *Objective Bayesian* methods) applies symmetry to the underlying measure space which, as we note in Appendix \[sec:cum\_dist\], may be very different from the space of observed magnitudes. This approach is concerned not directly with families of distributions, but with the particular distribution defined by a reference measure. We will briefly summarize the overlaps and differences of these methods. Objective Bayesian methods, initiated by Jeffries @jeffries57scientific but given the interpretation of objectivity largely by Jaynes @jaynes68prior, recognize that the reference measure $m_y$ in a relative entropy—beyond being needed to make logarithms well-defined and independent of change of variables—may reflect information about measurement scales. By ensuring that the reference measure is consistent with known symmetries of the phenomenon under study (which are not generally expressed within particular sample observations), Objective Bayesian methods seek to systematize the entire maximum-entropy procedure. This use of the reference measure is consistent with our treatment of measurement, though by itself it is more limited, as we discuss in Ref. @frank10measurement, and it may also be misleading in cases @seidenfeld79protect. In the context of the present discussion, the most important limitation of Objective Bayesian methods is that they select properties of a single distribution $m_y$, rather than properties of a family. Our approach broadens the class of symmetries that can be considered, beyond those available to Objective Bayesian methods, as discussed in Sec. 7 of Ref. @frank10measurement. Through the measurement function, it relates a potentially nonlinear contour of deformations of measured magnitudes to a linear transformation within the affine group that exists for general maximum-entropy problems. We have embedded distributions within a hierarchy by using the two-parameter freedom of the affine group to provide a range of responses of information to the change in the scale of measurement. The Morris classification of distributions in relation to maximum entropy ------------------------------------------------------------------------- In a pair of papers in 1982 and 1983 @morris82natural [@morris83natural], Carl Morris proposed another classification system for probability distributions, which overlaps both with Objective Bayesian methods and with our approach. Like our method, Morris's concerns families of probability distributions generated by a change in constraint or measurement scale. Like all of the approaches we have mentioned, Morris's system uses relative entropy in a conventional maximization framework. That system differs from ours in using only a linear constraint on what Morris terms the *natural observation*, and obtaining nonlinear dependence on that constraint through a second boundary condition placed on entropy. The Morris system blends interesting elements of Pearson's restrictions on analytic structure, our use of symmetry, and the Objective Bayesian concern with the reference measure, as follows: Morris considers distribution families that are invariant under offset and rescaling of the natural observation, which Morris labels $X$, and which is analogous to using a coordinate system that is always linear in our $f_y$. His classification therefore does not not invoke any explicit representation of the symmetries inherent in differing measurement systems. In order to encompass distributions that are not simply exponential in the values $x$ (taken by the observation $X$), he instead restricts the form of the reference measure in a relative entropy, analogous to our $m_y$. Unlike Objective Bayesian methods, however, this restriction does not come from the direct action of a symmetry on the reference measure, but rather from the form of the relative entropy across the family of distributions produced by scale change. The classification system of Ref. @morris82natural derives from the cumulant-generating function and the relation between the variance and the mean as the parameter in this generating function is shifted. The distributions that define the cumulant-generating function constitute what Morris calls *natural exponential families* (NEF), and the dependence of variance on mean within these families is restricted in his system to be an exact quadratic polynomial. The resulting subclass of distributions within the NEF class is termed QVF (for *quadratic variance function*). The mean-variance relation that defines the NEF-QVF distributions is preserved under offset and rescaling of the natural observation, and under convolution. Therefore, the distributions in this class would be expected to arise frequently in problems of aggregation. We show in this appendix that the QVF condition is equivalent to the requirement that the relative entropy over a family has the form (up to analytic continuation) of a Kullback-Leibler divergence. The analytic continuation is determined by the roots of the quadratic variance polynomial, and these roots in turn have a relation to the roots for log-probability in the Pearson system. The distributions selected by Morris's criterion are either bounded, or have exponential or faster decay in their tails. We show that, when they are classified according to their analytic structure, they are in fact either interior members or degenerate limits of only two families of distributions: One family of continuous-valued distributions is associated with complex-conjugate roots of the variance function, and a complex analytic continuation of the Kullback-Leibler form for relative entropy. A second family of discrete-valued distributions is associated with real-valued roots, and real-valued continuations of the Kullback-Leibler relative entropy. In this sense, the Morris classification shows that six important distribution families are in fact selected by a single set of invariances—of these, the offset and scale invariances are instances of our linear measurement rescaling. These selected families are therefore very commonly observed, but also rather tightly restricted. Preservation of a functional class under convolution is similar to the criterion leading to the extreme-value or Lévy distributions, as we have discussed in the main text, and is therefore one of many forms of measurement invariance that may be considered. Here we will re-formulate the Morris criterion and its solutions within a standard framework of maximum entropy. We will show that the role of the reference measure in a relative entropy is equivalent to that of a *second* observed quantity, which will generally be linearly independent of the natural observation $X$. Scale change of the natural observation defines what is known as an expansion path, which consists of the distributions within an exponential family. The second observed quantity, associated with the reference measure, is given a gradient constraint rather than a value constraint. It is through the interaction of these two constraints that nonlinear dependence on $x$ is obtained in the log-probability. At the end of the Appendix we mention a relation between the Morris system and the Pearson system based on the log-probability. When the Morris QVF criterion is expressed as a formal constraint on entropy, this form is imposed on the leading terms of log-probability by the large-deviations property of cumulant-generating functions. Definition of the natural exponential families ---------------------------------------------- The NEF distributions are defined in relation to the cumulant-generating function, which arises naturally in the method of maximum entropy. The most direct way to re-formulate the original presentation of Refs. @morris82natural [@morris83natural] in terms of maximum entropy is to assume a (Shannon-type) entropy in a higher-dimensional state space than the univariate space of the natural observation $X$. The high-dimensional states have non-uniform density when they are projected onto the one dimension in which the probability distribution varies. Once a Lagrangian is defined from this initial re-formulation, it becomes easy to re-interpret the density of states as a reference measure in a relative entropy (and the latter interpretation is more general). The cumulant-generating function is then the Legendre transform of this relative entropy. We develop the two interpretations in order, to connect the derivations of Refs. @morris82natural [@morris83natural] systematically to the formulation we use in the main text. ### The Stieltjes measure as a density of states Ref. @morris82natural introduces a Stieltjes measure ${{\hbox{\rm d}}}F \! \left( x \right)$, and an initial probability distribution $P_0$ associated with this measure, defined by $$P_0 \! \left( X \in A \right) = \int_A {{\hbox{\rm d}}}F \! \left( x \right) , \label{eq:Morris_pdf_0}$$ for an arbitrary set $A$ in the range of $x$. With respect to this original probability measure, Morris introduces the exponential families in terms of a probability mass function $$\phi \! \left( x \mid \theta \right) = e^{ x \theta - \psi \left( \theta \right) } , \label{eq:Morris_solution}$$ which multiplicatively weights the original measure ${{\hbox{\rm d}}}F \! \left( x \right)$. The normalizing constant $\psi \! \left( \theta \right)$ in Eq. (\[eq:Morris\_solution\]) is the cumulant-generating function, given by $$e^{ \psi \left( \theta \right) } \equiv \int {{\hbox{\rm d}}}F \! \left( x \right) e^{x \theta} . \label{eq:Morris_CGF}$$ The NEF distributions are the normalized versions of the distributions that define the cumulant-generating function. In the original Stieltjes measure, the probabilities defined from these distributions are $$P \! \left( X \in A \right) = \int_A {{\hbox{\rm d}}}F \! \left( x \right) e^{ x \theta - \psi \left( \theta \right) } . \label{eq:Morris_pdf}$$ With respect to the measure ${{\hbox{\rm d}}}F \! \left( x \right)$, we may obtain the solutions (\[eq:Morris\_solution\]) by extremizing the Lagrangian $$\mathcal{L} = - \int {{\hbox{\rm d}}}F \! \left( x \right) \, \phi \! \left( x \right) \log \phi \! \left( x \right) + \theta \left( \int {{\hbox{\rm d}}}F \! \left( x \right) \, \phi \! \left( x \right) x - \mu \right) - \kappa \left( \int {{\hbox{\rm d}}}F \! \left( x \right) \, \phi \! \left( x \right) - 1 \right) \label{eq:Lagrange_Morris}$$ over its natural argument $\phi \! \left( x \right)$ and the Lagrange multipliers $\theta$ and $\kappa$. Here we have replaced the notation $\lambda$ from the text with Morris's $\theta$ for ease of reference. From its role as a normalization constant, the multiplier $\kappa$ must evaluate to the cumulant-generating function $\psi \! \left( \theta \right)$ on solutions. Lagrangian problems of this form arise frequently in systems where a high-dimensional state space is projected down onto a single coordinate $x$, which is the only observed property on which distributions depend. The Lagrangian (\[eq:Lagrange\_Morris\]) effectively treats $\phi \! \left( x \right)$ as the ratio of a probability density to a *uniform* reference measure on the original high-dimensional space. The Stieltjes measure ${{\hbox{\rm d}}}F \! \left( x \right)$ is the marginal projection of the original measure onto the coordinate $x$, and the derivative ${{\hbox{\rm d}}}F / {{\hbox{\rm d}}}x$ is known as the *density of states*. (${{\hbox{\rm d}}}F \! \left( x \right)$ need not be smooth, and ${{\hbox{\rm d}}}F / {{\hbox{\rm d}}}x$ may readily be a non-continuous distribution, such as a sum of Dirac $\delta$-functions, representing a discrete rather than continuous probability density). The entropy in this formulation appears as a standard Shannon entropy (equivalent to a relative entropy with a uniform reference measure) in the high-dimensional coordinates. It evaluates to the Legendre transform of the cumulant-generating function, $$\begin{aligned} S \! \left( \mu \! \left( \theta \right) \right) & \equiv & - \int {{\hbox{\rm d}}}F \! \left( x \right) \, p \! \left( x \mid \theta \right) \log p \! \left( x \mid \theta \right) \nonumber \\ & = & - \theta \mu \! \left( \theta \right) + \psi \! \left( \theta \right) , \label{eq:Morris_ent_eval}\end{aligned}$$ in which $\mu \! \left( \theta \right)$ is the mean value in the distribution $p \! \left( x \mid \theta \right)$. $\theta$ is the natural argument of $\psi$, while $\mu$ from the variational problem is the natural argument of $S$. Therefore it is usual to write this Legendre transform pair as $$\begin{aligned} \psi \! \left( \theta \right) & = & { \left. S \! \left( \mu \right) - \mu \frac{{{\hbox{\rm d}}}S}{{{\hbox{\rm
{'source': 'arXiv_out/1010'}