Generalised Measures of Multivariate Information Content
Conor Finn Joseph T. Lizier


Adversarial Machine Learning Attack on Modulation Classification
    Muhammad Usama1,
Muhammad Asim1,
Junaid Qadir1,
Ala Al-Fuqaha2,
Muhammad Ali Imran3

1Information Technology University, Lahore, Punjab, Pakistan.
2Hamad Bin Khalifa University, Qatar
3University of Glasgow, Scotland, UK

 
Email: 1(muhammad.usama, msee16001, junaid.qadir)@itu.edu.pk, 2aalfuqaha@hbku.edu.qa,
3muhammad.imran@glasgow.ac.uk

    December 30, 2023
========================================================================================================================================================================================================================================================================================================================================================




  The entropy of a pair of random variables is commonly depicted using a Venn diagram.  This
  representation is potentially misleading, however, since the multivariate mutual information can
  be negative.  This paper presents new measures of multivariate information content that can be
  accurately depicted using Venn diagrams for any number of random variables.  These measures
  complement the existing measures of multivariate mutual information and are constructed by
  considering the algebraic structure of information sharing.  It is shown that the distinct ways in
  which a set of marginal observers can share their information with a non-observing third party
  corresponds to the elements of a free distributive lattice.  The redundancy lattice from partial
  information decomposition is then subsequently and independently derived by combining the
  algebraic structures of joint and shared information content.

  



§ INTRODUCTION





For any pair of random variables X and Y, the entropy H satisfies the inequality

    H(X) + H(Y) ≥ H(X,Y) ≥ H(X),  H(Y) ≥ 0.

From this inequality, it is easy to see that the conditional entropies and mutual information are
non-negative,

    H(X|Y)    = H(X,Y) - H(Y) ≥ 0, 
    
      H(Y|X)    = H(X,Y) - H(X) ≥ 0, 
    
      I(X;Y)    = H(X) + H(Y) - H(X,Y) ≥ 0.

For any pair of sets A and B, a measure μ satisfies the inequality

    μ(A) + μ(B) ≥μ(A ∪ B) ≥μ(A), μ(B) ≥ 0,

which follows from the non-negativity of measure on the relative complements and the
intersection,

  μ(A B)    = μ(A ∪B) - μ(B) ≥0  

  μ(B A)    = μ(A ∪B) - μ(A) ≥0  

  μ(A ∩B)    = μ(A) + μ(B) - μ(A ∪B) ≥0 .  

Although the entropy is not itself a measure, several authors have noted the entropy is analogous to
measure in this regard <cit.>.  Indeed, it is this analogy which provides the justification for the typical depiction
of a pair of entropies using Venn diagrams, i.e. analogy.  Nevertheless, <cit.>
notes that this representation is misleading for at least two reasons: Firstly, since the measure on
the intersection μ(A ∩ B) is a measure on a set, it gives the false impression that the
mutual information I(X;Y) is the entropy of some intersection between the random variables.
Secondly, it might lead one to believe that this analogy can be generalised beyond two variables.
However, the analogy does not generalise beyond two variables since the multivariate mutual
information between three random variables,

    I(X;Y;Z)    = H(X) + H(Y) + H(Z) - H(X,Y) - H(X,Z) - H(Y,Z) + H(Z,Y,Z),

is not non-negative <cit.>, and hence is not analogous to measure on the
triple intersection μ(A ∩ B ∩ C) <cit.>.  Indeed, this “unfortunate”
property led Cover and Thomas to conclude that “there isn't really a notion of mutual information
common to three random variables” <cit.>.  Consequently, <cit.>
recommends against depicting the entropy of three or more variables using a Venn diagram, i.e.analogy, unless one is aware of these issues with this representation. 

However, <cit.> showed that there is an analogy between entropy and signed measure
that is valid for an arbitrary number of random variables.  To do this, Yeung defined a signed
measure on a suitably constructed sigma-field that is uniquely determined by the joint entropies of
the random variables involved. This correspondence enables one to establish information-theoretic
identities from measure-theoretic identities.  Thus, Venn diagrams can be used to represent the
entropy of three or more variables provided one is aware that the certain overlapping areas may
correspond to negative quantities.  Moreover, the multivariate mutual information is useful both as
summary quantity and for manipulating information-theoretic identities provided one is mindful it
may have “no intuitive meaning” <cit.>.

In this paper, we introduce new measures of multivariate information that are analogous to measures
upon sets and maintain their operational meaning when considering an arbitrary number of variables.
These new measures complement the existing measures of multivariate mutual information, and will be
constructed by considering the distinct ways in which a set of marginal observers might share their
information with a non-observing third party.  In mic, we will discuss the existing
measures of information content in terms of a set of individuals who each have different knowledge
about a joint realisation from a pair of random variables.  Then in marginal, we will
discuss how these individuals can share their information with a non-observing third party, and
derive the functional form of this individual's information.  In synergy, we relate this
new measure of information content back to the mutual information.
Sections <ref>–<ref> will then generalise the arguments of
Sections <ref> and <ref> to consider an arbitrary number of observers.
Finally, in info, we will discuss how these new measures can be combined to define new
measures of mutual information.




§ MUTUAL INFORMATION CONTENT



Suppose that Alice and Bob are separately observing some process and let the discrete random
variables X and Y represent their respective observations.  Say that Johnny is a third
individual who can simultaneously make the same observations as Alice and Bob such that his
observations are given by the joint variable (X,Y).  When a realisation (x,y) occurs, Alice's
information is given by the information content <cit.>,

    h(x) = - log p_X(x) ≥ 0,

where p_X(x) is the probability mass of the realisation x of variable X computed from the
probabilty distribution p_X.  Likewise, Bob's information is given by the information content
h(y), while Johnny's information is by the joint information content h(x,y)=-log p_XY(x,y).
The information that Alice can expect to gain from an observation is given by the entropy,

    H(X) = E_X[h(x)] ≥ 0,

where E_X represents an expectation value over realisations of the variable X.
Similarly, Bob's expected information gain is given by the entropy H(X) and Johnny's
expected information is given by the joint entropy H(X,Y)=E_XY [h(x,y)].  Clearly, for
any realisation, Johnny has at least as much information as either Alice or Bob,

    h(x,y) ≥ h(x), h(y) ≥ 0.

The conditional information content can be used to quantify how much more information Johnny has
relative to either Alice or Bob, respectively,

     h(x|y)       = h(x,y) - h(y)       ≥0, 

     h(y|x)       = h(x,y) - h(x)       ≥0. 

Similarly, we can quantify how much more information Johnny expects to get compared to either Alice
or Bob via the conditional entropies,

     H(X|Y)       = E_XY [h(x|y)]       ≥0,  

     H(Y|X)       = E_XY [h(y|x)]       ≥0. 


Now consider a fourth individual who does not directly observe the process, but with whom Alice and
Bob share their knowledge.  To be explicit, we are considering the situation whereby this individual
knows that the joint realisation (x,y) has occurred and knows the marginal distributions p_X and
p_Y, but does not know the joint distribution p_XY.  How much information does this individual
obtain from the shared marginal knowledge provided by Alice and Bob?  The answer to this question
will be provided in marginal, but for now let us consider a simplified version of this
problem.  Suppose that such an individual, whom we will call Indiana, assumes that Alice's
observations are independent of Bob's observations.  In terms of the probabilities, this means that
Indy believes that the joint probability p_XY(x,y) is equal to the product probability
p_X × Y(x,y) = p_X(x) p_Y(y), while in terms of information, this assumption leads Indiana
to believe that her information is given by the independent information content h(x) + h(y).
Moreover, the information that Indiana expects to gain from any one realisation is given by
H(X)+H(Y).

Let us now compare how much information Indiana believes that she has compared to our other
observers.  For every realisation, Indiana believes that she has at least as much information as
either Alice or Bob,

    h(x) + h(y) ≥ h(x), h(y) ≥ 0.

Since Indy knows what both Alice and Bob know individually, it is hardly surprising that she always
has at least as much information as either Alice or Bob.  The comparison between Indiana and Johnny,
however, is not so straightforward—there is no inequality that requires the information content of
the joint realisation to be less than the information content of the independent realisations, or
vice versa.  Consequently, the difference between the information that Indiana thinks she has and
Johnny's information, i.e. the mutual information content between a pair of realisations,

    i(x;y) = h(x) + h(y) - h(x,y) = logp_XY(x,y)/p_X(x) p_Y(y),

is not non-negative <cit.>.  (This function goes by several different names including the
pointwise mutual information, the information density <cit.> or simply the mutual
information <cit.>.)  Thus, similar to how it is potentially misleading to depict the
entropy of three of more variables using a Venn diagram, representing the information content of two
variables using a Venn diagram is somewhat dubious (see analogy_pw).



Since Johnny knows the joint distribution p_XY, while Indiana only knows the marginal
distributions p_X(x) and p_Y(y), we might expect that Indiana should never have more information
than Johnny.  However, Indiana's assumed information is based upon the belief that Alice's
observations X are independent of Bob's observations Y, which leads Indiana to overestimate her
information on average.  Indeed, Indiana is so optimistic that the information she expects to get upper
bounds the information that Johnny can expect to get,

    H(X) + H(Y) ≥ H(X,Y) ≥ 0.

Thus, despite the fact that Indiana can have less information than Johnny for certain
realisations—i.e. despite the fact that the mutual information content is not non-negative—the
mutual information in expectation is non-negative,

    I(X;Y) =  H(X) + H(Y) - H(X,Y) = E_XY[i(x;y)] ≥ 0.

Crucially, and in contrast to the information content info_content and entropy entropy,
the non-negativity of the mutual information does not follow directly from the non-negativity of the
mutual information content mutual_info_content, but rather must be proved separately.
(Typically, this is done by showing that the mutual information can be written as a Kullback-Leibler
divergence which is non-negative by Jensen's inequality, e.g. see <cit.>.)  Thus, not
only does Indiana potentially have more information than Johnny for certain realisations, but on
average we expect Indiana to have more information than Johnny.  Of course, by assuming Alice's
observations are independent of Bob's observations, Indiana is overestimating her information.
Thus, in the next section, we will consider the situation whereby one does not make this assumption.




§ MARGINAL INFORMATION SHARING



Suppose that Eve is another individual who, similar to Indiana, does not make any direct
observations, but with whom both Alice and Bob share their knowledge; i.e. Eve knows the joint
realisation (x,y) has occurred and knows the marginal distributions p_X and p_Y, but does not
know the joint distribution p_XY.  Furthermore, suppose that Eve is more conservative than
Indiana and does not assume that Alice's observations are independent of Bob's observations—how
much information does Eve have for any one realisation?

It seems clear that Eve's information should always satisfy the following two requirements. Firstly,
since Alice and Bob both share their knowledge with Eve, she should have at least as much
information as either of them have individually.  Secondly, since Eve has less knowledge than
Johnny, she should have no more information than Johnny; i.e. in contrast to Indy, Eve should never
have more information than Johnny.  As the following theorem shows, these two requirements uniquely
determine the functional form of Eve's information:

  
  The unique function h(x ⊔ y) of p_X(x) and p_Y(y) that satisfies
  h(x,y) ≥ h(x ⊔ y) ≥ h(x), h(y) ≥ 0 for all p_XY(x,y) is
  
    h(x ⊔ y) = max(h(x),h(y)) ≥ 0.



  Clearly, the function is lower bounded by max(h(x),h(y)).  The upper bound is given by
  the minimum possible h(x,y) which corresponds to the maximum allowed p_XY(x,y).  For any
  p_X(x) and p_Y(y), the maximum allowed p_XY(x,y) is min(p_X(x),p_Y(y)), which
  corresponds to h(x,y)=max(h(x),h(y)).

Eve's information is given by the maximum of Alice's and Bob's information, or the information
content of the most surprising marginal realisation.  Although we have defined Eve's information by
requiring it to be no greater than Johnny's information, it is also clear that Eve also has no more
information than Indiana.  As such, Eve's information satisfies the inequality

    h(x) + h(y) ≥ h(x ⊔ y) ≥ h(x), h(y) ≥ 0,

which is analogous to the inequality measure_inequalities satisfied by measure.  Hence, as
pre-empted by the notation (and will be further justified in generalised), Eve's
information will be referred to as the union information content.  The union information
content is the maximum possible information that Eve can get from knowing what Alice and Bob
know—it quantifies the information provided by a joint event (x,y) when one knows the marginal
distributions p_X and p_Y, but does not know nor make any assumptions about the joint
distribution p_XY.

Similar to how the conditional information contents cond_entropy_XY and cond_entropy_YX
enables us to quantify how much more information Johnny has relative to either Alice or Bob, the
inequality union_inequality enables us to quantify how much information Eve gets from Alice
relative to Bob and vice versa, respectively,

     h(x y)       = h(x ⊔y) - h(y)       = max(h(x) - h(y),0)       ≥0,

     h(y x)       = h(x ⊔y) - h(x)       = max(0,h(y) - h(x))       ≥0.

These non-negative functions are analogous to measure on the relative complements of a pair of sets
and will be called the unique information content from x relative to y, and vice versa
respectively.  It is easy to see that, since Eve's information is either equal to Alice's or Bob's
information (or both), at least one of these two functions must be equal to zero.  

The inequality union_inequality also enables us to quantify how much more information Indiana
has relative to Eve.  Since Indiana's assumed information is given by the sum of Alice's and Bob's
information while Eve's information is given by the maximum of Alice's and Bob's information, the
difference between the two is given by the minimum of Alice's and Bob's information,

    h(x ⊓ y) = h(x) + h(y) - h(x ⊔ y) = h(x) + h(y) - max(h(x), h(y) ) 
      = min(h(x), h(y)) ≥ 0.

In contrast to the comparison between Indiana and Johnny, i.e. the mutual information content
mutual_info_content, the comparison between Indiana and Eve is non-negative.  As such, this
function is analogous to measure on the intersection of two sets and hence will be referred to as
the intersection information content.  The intersection information content is the minimum
possible information that Eve could have gotten from knowing either what Alice or Bob know, and is
given by the information content of the least surprising marginal realisation.

Finally, from union_info_content and uni_ic_x–max_min_two, it is not difficult to
see that Eve's information can be decomposed into the information that could have been obtained from
either Alice or Bob, the unique information from Alice relative to Bob and the unique information
from Bob relative to Alice,

    h(x ⊔ y) = h(x ⊓ y) + h(x  y) + h(y  x).

Of course, as already discussed, at least one of these unique information contents must be zero.
union_info depicts this decomposition for some realisation whereby Alice's information h(x)
is greater than Bob's information h(y).


To summarise thus far, both Alice and Bob share their information with Indiana and Eve, who then
each interpret this information in a different way.  By comparing analogy_pw and
union_info, we can easily contrast their distinct perspectives.  Eve is more conservative than
Indiana and assumes that she has gotten as little information as she could possibly have gotten from
knowing what Alice and Bob know; this is given by the maximum from Alice's and Bob's information, or
is the information content associated with the most surprising marginal realisation observed by
Alice and Bob.  In effect, Eve's conservative approach means that she pessimistically assumes that
the information provided by the least surprising marginal realisation was already provided by the
most surprising marginal realisation.  In contrast, Indiana optimistically assumes that the
information provided by the least surprising marginal realisation is independent of the information
provided by the most surprising marginal realisation.






Let us now consider the information that Eve expects to get from a single realisation,

    H(X ⊔ Y) = E_XY[h(x ⊔ y)] ≥ 0.

This function will be called the union entropy, and quantifies the expected surprise of the most
surprising realisation from either X or Y.  Similar to how the non-negativity of the entropy
entropy follows from the non-negativity of the information content info_content, the
non-negativity of the union entropy union_entropy follows directly from the non-negativity of
the union information content union_info_content.  Since the expectation value is monotonic,
and since the union information content satisfies the inequality union_inequality, we get that
the union entropy satisfies

    H(X) + H(Y) ≥ H(X ⊔ Y) ≥ H(X), H(Y) ≥ 0,

and hence is also analogous to measure on the union of two sets.

Using this inequality, we can quantify how much more information Eve expects to get from Alice
relative to Bob, or vice versa respectively,

     H(X Y)      =H(X ⊔Y) - H(Y)      =E_XY [h(x y)]      ≥0,

     H(Y X)      =H(X ⊔Y) - H(X)      =E_XY [h(y x)]      ≥0,

These functions are also analogous to measure on the relative complements of a pair of sets and
hence will be called the unique entropy from X relative to Y, and vice versa respectively.
Crucially, and in contrast to uni_ic_x and uni_ic_y, both of these quantities can be
simultaneously non-zero; although Alice might observe the most surprising event in one joint
realisation, Bob might observe the most surprising event in another and hence both functions can be
simultaneously non-zero.

Now consider how much more information Indiana expects to get relative to Eve,

    H(X ⊓ Y) = H(X) + H(Y) - H(X ⊔ Y) = E_XY[h(x ⊓ y)] ≥ 0.

This function is also analogous to measure on the intersection of two sets function will be called
the intersection entropy.  In contrast to the mutual information mutual_info, since the
intersection information content max_min_two is non-negative, we do not require an additional
proof to show that the intersection entropy is non-negative.

Finally, similar to decomp_union, we can decompose Eve's expected information into the
following components,

    H(X ⊔ Y) = H(X ⊓ Y) + H(X  Y) + H(Y  X).

It is important to reiterate that, in contrast to decomp_union, there is nothing which requires
either of the two unique entropies to be zero.  Thus, as shown in union_info, the Venn diagram
which represents the union and intersection entropy differs from that which represents the union
information content.




§ SYNERGISTIC INFORMATION CONTENT





As we discussed at the beginning of the previous section, and as we required in
Theorem <ref>, one of the defining features of Eve's information is that it is never greater
than Johnny's information,

    h(x,y) ≥ h(x ⊔ y).

Thus, we can compare how much more information Johnny has relative to Eve,

    h(x ⊕ y) = h(x,y) - h(x ⊔ y)
        = h(x,y) - max(h(x), h(y) )
        = min(h(y|x), h(x|y)) ≥ 0.

This non-negative function will be called the synergistic information content, and it
quantifies how much more information one gets from knowing the joint probability p_XY(x,y)
relative to merely knowing the marginal probabilities p_X(x) and p_Y(y).
synergistic_info_content shows how this relationship can represented using a Venn diagram.  Of
course, by this definition, Johnny's information is equal to the union information content plus the
synergistic information content, and hence, by using decomp_union, we can decompose Johnny's
information into the intersection information content, the unique information contents, and the
synergistic information contents,

    h(x,y) = h(x ⊔ y) + h(x ⊕ y) = h(x ⊓ y) + h(x  y) + h(y  x) + h(x ⊕ y).

This decomposition can be seen in synergistic_info_content, although it is important to recall
that at least one of h(x  y) and h(y  x) must be equal to zero.  In a similar manner,
the extra information that Johnny has relative to Bob cond_info_content_y can be decomposed
into the unique information content from Alice and the synergistic information content, and vice
versa for the extra information that Johnny has relative to Alice cond_info_content_x,

    h(x|y)    = h(x  y) + h(x ⊕ y), 
    
      h(y|x)    = h(y  x) + h(x ⊕ y).


Now recall that the mutual information content mutual_info_content is given by Indiana's
information minus Johnny's information.  By replacing Johnny's information with the union
information content plus the synergistic information content via synergistic_ic and rearranging
using max_min_two, we get that the mutual information content is equal to the intersection
information content minus the synergistic information content,

    i(x;y) = h(x) + h(y) - h(x,y) = h(x) + h(y) - h(x ⊔ y) - h(x ⊕ y)
        =  h(x ⊓ y) - h(x ⊕ y).

Indeed, this relationship can be identified in synergistic_info_content.  Clearly, the mutual
information content is negative whenever the synergistic information content is greater than the
intersection information content.  From this perspective, the mutual information content can be
negative because there is nothing to suggest that the synergistic information content should be no
greater than the intersection information content.  In other words, the additional surprise
associated with knowing p_XY(x,y) relative to merely knowing p_X(x) and p_Y(y) can exceed
the surprise of the least surprising marginal realisation.

Let us now quantify how much more information Johnny expects to get relative to Eve,

    H(X ⊕ Y) = E_XY[h(x ⊕ y)] = H(X,Y) - H(X ⊔ Y) ≥ 0,

which we will call the synergistic entropy.  Crucially, although the synergistic information content
is given by the minimum of the two conditional information contents, the synergistic entropy does
not in general equal one of the two the conditional entropies.  This is because, although Alice
might observe the most surprising event in one joint realisation such that the synergistic
information content is equal to Bob's information given Alice's information, Bob might observe the
most surprising event in another realisation such that the synergistic information content is equal
to Alice's information given Bob's information for that particular realisation.  Thus, the
synergistic entropy does not equal the conditional entropy for the same reason that unique entropies
uni_e_x and uni_e_y can be simultaneously non-zero.



With the definition of synergistic entropy, it is not difficult to show that, similar to
bivar_decomp, the joint entropy can be decomposed into the following components,

    H(X,Y) = H(X ⊔ Y) + H(X ⊕ Y) = H(X ⊓ Y) + H(X  Y) + H(Y  X) + H(X ⊕ Y).

synergistic_entropy depicts this decomposition using a Venn diagram, and shows how the union
entropy from union_info is related to the joint entropy H(X,Y).  Likewise, similar to
cond_decomp_x and cond_decomp_y, it is easy to see that conditional entropies can be
decomposed as follows,

    H(X|Y)    = H(X  Y) + H(X ⊕ Y), 
    
      H(Y|X)    = H(Y  X) + H(X ⊕ Y)

Finally, just like mi_diff, we can also show that the mutual information is equal to the
intersection entropy minus the synergistic entropy,

    I(X;Y) =  H(X ⊓ Y) - H(X ⊕ Y) ≥ 0.

Although there is nothing to suggest that the synergistic information content must be no greater
than the intersection information content, we know that the synergistic entropy must be no greater
than the intersection entropy because I(X;Y) ≥ 0.  In other words, the expected difference
between the surprise of the joint realisation and the most surprising marginal realisation cannot
exceed the expected surprise of the least surprising realisation.




§ PROPERTIES OF THE UNION AND INTERSECTION INFORMATION CONTENT



Theorem <ref> determined the function form of Eve's information when Alice and Bob share
their knowledge with her.  We now wish to generalise this result to consider the situation whereby
an arbitrary number of marginal observers share their information with Eve.  Rather than try to
directly determine the functional form, however, we will proceed by considering the algebraic
structure of shared marginal information.  

If Alice and Bob observe the same realisation x such that they have the same information h(x),
then upon sharing we would intuitively expect Eve to have the same information h(x).  Similarly,
the minimum information that Eve could have received from either Alice or Bob should be the same
information h(x).  Since the maximum and minimum operators are idempotent, the union and
intersection information content both align with this intuition.

  
  The union and intersection information content are idempotent,
  
    h(x ⊔ x)    = h(x), 
    
        h(x ⊓ x)    = h(x).



It also seems reasonable to expect that Eve's information should not depend on the order in which
Alice and Bob share their information, and nor should the minimum information that Eve could have
received from either individual.  Again, since the maximum and minimum operators are commutative,
the union and intersection information content both align with our intuition.

  
  The union and intersection information content are commutative,
  
    h(x ⊔ y)    = h(y ⊔ x), 
    
        h(x ⊓ y)    = h(y ⊓ x).



Now suppose that Charlie is another individual who, just like Alice and Bob, is separately observing
some process, and let the random variable Z represent her observations.  Say that Dan is yet
another individual with whom, just like Eve, our observers can share their information. Intuitively,
it should not matter whether Alice, Bob and Charlie share their information directly with Eve, or
whether they share their information through Dan.  To be specific, Alice and Bob could share their
information with Dan such that his information is given by h(x ⊔ y), and then Charlie and Dan
could subsequently share their information with Eve such that her information is given by
h((x ⊔ y) ⊔ z ).  Similarly, Bob and Charlie could share their information with
Dan such that his information is given by h(y ⊔ z), and then Alice and Dan could subsequently
share their information with Eve such that her information is given by
h(x ⊔ (y ⊔ z)).  Alternatively, Alice, Bob and Charlie could entirely bypass Dan
and share their information directly with Eve such that her information is given by
h(x ⊔ y ⊔ z).  Since the maximum operator is associative, the union information content
is the same in all three cases and hence aligns with our intuition.  A similar argument can be made
to show that the intersection information content is also associative.

  
  The union and intersection information content are associative,
  
       h(x ⊔y ⊔z)    = h((x ⊔y) ⊔z )    = h(x ⊔(y ⊔z)), 

       h(x ⊓y ⊓z)    = h((x ⊓y) ⊓z )    = h(x ⊓(y ⊓z)),  
  


Suppose now that Alice and Bob share their information with Dan such the information that he could
have gotten from either Alice or Bob is given by h(x ⊓ y).  If Alice and Dan both share their
information with Eve, then Eve's information is given by

    h(x ⊔ (x ⊓ y)) = max(h(x),min(h(x),h(y))) = h(x),

and hence Bob's information has been absorbed by Alice's information.  Now suppose that Alice and
Bob share their information with Dan such his information is given by h(x ⊔ y).  If Alice and
Dan both share their information with Eve, then the information that Eve could have gotten from
either Alice or Dan is given by

    h(x ⊓ (x ⊔ y)) = min(h(x),max(h(x),h(y))) = h(x).

Again, Bob's information has been absorbed by Alice's information.  Both of these results are a
consequence of the fact that the maximum and minimum operators are connected to each other by the
absorption identity.

  
  The union and intersection information content are connected by absorption,
  
    h(x ⊔ (x ⊓ y))    = h(x), 
    
        h(x ⊓ (x ⊔ y))    = h(x).



Now say that Daniella is, just like Eve or Dan, an individual with whom our observers can share
their information.  Consider the following two cases: Firstly, suppose that Bob and Charlie share
their information with Dan such that the information that Dan could have gotten from either Bob or
Charlie is given by h(y ⊓ z).  If both Alice and Dan share their information with Eve, then
her information is given by h(x ⊔ (y ⊓ z)).  In the second case, suppose that
Alice and Bob share their information with Dan such that his information is given by
h(x ⊔ y), while Alice and Charlie simultaneously share their information with Daniella such
that her information is given by h(x ⊔ z).  If Dan and Daniella both share their information
with Eve, then the information that she could have gotten from either Dan or Daniella is then given
by h((x ⊔ y) ⊓ (x ⊔ z)).  In both cases, Eve has the same information since
the maximum operator is distributive,

    h(x ⊔ (y ⊓ z))    = max(h(x),min(h(y),h(z))) 
       =min(max(h(x),h(y)),max(h(x),h(z)))
          = h((x ⊔ y) ⊓ (x ⊔ z)).

Since the maximum and minimum operators are distributive over each other, regardless of whether Eve
gets Alice's information and Bob's or Charlie's information, or if Eve gets Alice's and Bob's
information or Alice's and Charlie's information, Eve has the same information. The same reasoning
can be applied to show that regardless of whether Eve gets Alice's information or Bob's and
Charlie's information, or if Eve gets Alice's or Bob's information and Alice's or Charlie's
information, Eve has the same information.

  
  The union and intersection information content are distribute over each other,
  
    h(x ⊔ (y ⊓ z))    = h((x ⊔ y) ⊓ (x ⊔ z)),
                                       
    
        h(x ⊓ (y ⊔ z))    = h((x ⊓ y) ⊔ (x ⊓ z)).





Now consider a set of n individuals and let X = { X_1, X_2, …, X_n} be the joint
random variable that represents their observations.  Suppose that these individuals together observe
the joint realisation x = { x_1, x_2, …, x_n} from X.  By Property <ref> and the general associativity theorem, it is clear that Eve's information is given
by

    h(x_1 ⊔ x_2 ⊔…⊔ x_n ) = max(h(x_1), h(x_2), …, h(x_n) ) ≥ 0,

while the minimum information that Eve could have gotten from any individual observer is given by

    h(x_1 ⊓ x_2 ⊓…⊓ x_n ) = min(h(x_1), h(x_2), …, h(x_n) ) ≥ 0.

This accounts for the situation whereby n marginal observers directly share their information with
Eve, and could clearly be considered for any subset S of the observers X.  We now wish
to consider all of the distinct ways that these marginal observers can share their information
indirectly with Eve.  As the following theorem shows, Properties <ref>–<ref> completely characterise the unique methods of marginal information sharing.


  The marginal information contents form a join semi-lattices ⟨x,h( ⊔ )⟩
  under the max operator.  Separately, the marginal information contents form a meet semi-lattice
  ⟨x,h( ⊓ )⟩ under the min operator.


  Properties <ref>–<ref> completely characterise semi-lattices
  <cit.>.



  The marginal information contents form a distributive lattice
  ⟨x,h( ⊔ ),h( ⊓ )⟩ under the max and min operators.


  From Property <ref>, we have that the semi-lattices
  ⟨x,h( ⊔ )⟩ and ⟨x,h( ⊓ )⟩ are connected by
  absorption and hence form a lattice ⟨x,h( ⊔ ),h( ⊓ )⟩.  By
  Property <ref> this is a distributive lattice <cit.>.


Each way that a set of n observers can share their information with Eve such that she has distinct
information corresponds to an element in partially ordered set, or more specifically the free
distributive lattice on n generators <cit.>.  distributive_lattices shows the
free distributive lattices generated by n=2 and n=3 observers.  The number of elements in this
lattice given by the (n)-th Dedekind number <cit.> (see also <cit.>).
By the fundamental theorem of distributive lattices (or Birkhoff's representation theory), there is
isomorphism between the union information content and set union, and between the intersection
information content and set intersection <cit.>.
It is this one-to-one correspondence that justifies our use of the terms union and intersection
information content for n variables in general.  Every identity that holds in a lattice of sets
will have a corresponding identity in this distributive lattice of information contents.
distributive_lattices also depicts the sets which correspond to each term in the lattice of
information contents. Just as the cardinality of sets is non-decreasing as we consider moving up
through the various terms in a lattice of sets, Eve's information is non-decreasing as we moving up
through the various terms in the corresponding lattice of information contents.  In particular, we
can quantify the unique information content that Eve gets from one method of information sharing
relative to any other method that is lower in the lattice.

Every property of the union and intersection information content that we have considered thus far
has been directly inherited by the union and intersection entropy.  However, there is one final
property is not inherieted by the entropies.  If Alice and Bob share their information with Eve,
then Eve's information is given by either Alice's or Bob's information, and similar for the
information that Eve could have gotten from either Alice or Bob.  As the subsequent theorem shows,
this property enables us to greatly reduce the number of distinct terms in the distributive lattice
for information content since any partially ordered set with a connex relation forms a total order.


  
  The union and intersection information content are given by at least one of
  
    h(x ⊔ y) = h(x)  and h(x ⊓ y) = h(y),   or  
        h(x ⊔ y) = h(y)  and h(x ⊓ y) = h(x)






§ GENERALISED MARGINAL INFORMATION SHARING



We will now use Properties <ref>–<ref> to generalise the results of
Theorem <ref> and marginal.


  The marginal information contents are a totally ordered set under the max and min operators.


  A totally ordered set is a partially ordered set with the connex
  property <cit.>.




total_order shows the totally ordered sets generated by n=2 and n=3 observers, and also
depicts the corresponding sets.  Although the number of distinct terms has been reduced, Eve's
information is still non-decreasing as we move up through terms of the totally ordered set.  If we
now compare how much unique information Eve gets from a given method of information sharing relative
to any other method of information sharing which is equal or lower in the totally ordered set, then
we obtain a result which generalises uni_ic_x and uni_ic_y to consider more than two
observers.  Similarly, this total order enables us to generalise max_min_two using the
maximum-minimum identity <cit.>, which is a form of the principle of
inclusion-exclusion <cit.> for a totally ordered set,

    h(x_1 ⊓ x_2 ⊓…⊓ x_n) 
         = min(h(x_1), h(x_2), …, h(x_n)) 
       = ∑_k=1^n (-1)^k-1∑_S⊆X
    |S|=k max(h(s_1), h(s_2), …, h(s_k)) 
       = ∑_k=1^n (-1)^k-1∑_S⊆X
    |S|=k 
        h(s_1 ⊔ s_2 ⊔…⊔ s_k),

or conversely,

    h(x_1 ⊔ x_2 ⊔…⊔ x_n)
        = ∑_k=1^n (-1)^k-1∑_S⊆X
    
        |S|=k  h(s_1 ⊓ s_2 ⊓…⊓ s_k).


Now that we have generalised the union and intersection information content, similar to
marginal, let us now consider taking the expectation value for each term in the
distributive lattice.  For every joint realisation x from X, there is a corresponding
distributive lattice of information contents.  Hence, similar to union_entropy and
uni_e_x–intersection_entropy, we can consider taking the expectation value of each term
in the lattice over all realisations.  Since the expectation is a linear operator, this yields a set
of entropies that are also idempotent, commutative, associative, absorptive and distributive, only
now over the random variables from X.  Thus, the information that Eve expects to gain from a
single realisation for a particular method of information sharing also corresponds to a term in a
free distributive lattice generated by n.  This distributive lattice for entropies can be seen in
distributive_lattices by replacing x, y, z and h with X, Y, Z and H
respectively.

Crucially, however, Property <ref> does not hold for the entropies—it is not true that
Eve's expected information H(X ⊔ Y) is given by either Alice's expected information H(X) or
Bob's expected information H(Y).  Thus, despite the fact that the distributive lattice of
information content can be reduced to a total order, the distributive lattice of entropies remains
partially ordered.  Although the information contents are totally ordered for every realisation,
this order is not in general the same for every realisation.  Consequently, when taking the
expectation value across many realisations to yield the corresponding entropies, the total order is
not maintained, and hence we are left with a partially ordered set of entropies.  Indeed, we already
saw the consequences of this result in union_info whereby Alice's and Bob's information
content was totally ordered for any one realisation, but their expected information was partially
ordered.




§ MULTIVARIATE INFORMATION DECOMPOSITION



In synergy, we used the shared marginal information from marginal to decompose the
joint information content into four distinct components.  Our aim now is use the generalised notion
of shared information from the previous section to produce a generalised decomposition of the joint
information content.  To begin, suppose that Johnny observes the joint realisation (x,y,z) while
Alice, Bob and Charlie observe the marginal realisations x, y and z respectively, and say that
Alice, Bob and Charlie share their information with Eve such that her information is given by
h(x ⊔ y ⊔ z).  Clearly, Johnny has at least as much information as Eve,

    h(x, y, z) ≥ h(x ⊔ y ⊔ z).

Thus, we can compare how much more information Johnny has relative to Eve,

    h(x ⊕ y ⊕ z) = h(x,y,z) - h(x ⊔ y ⊔ z)
      = min( h(y,z|x), h(x,z|y), h(x,y|z) ) ≥ 0.

This non-negative function generalises the earlier definition of the synergistic information content
synergistic_ic such that it now quantifies how much information one gets from knowing the joint
probability p_XYZ(x,y,z) relative to merely knowing the three marginal probabilities p_X(x),
p_Y(y) and p_Z(z).  union_three shows how this relationship can be represented using a
Venn diagram.



Now consider three more observers, Joan, Jonas, and Joanna, who observer the joint marginal
realisations (x,y), (x,z) and (y,z), respectively.  Clearly, these additional observers
greatly increase the number of distinct ways in which marginal information might be shared with Eve.
For example, if Alice and Joanna share their information, then Eve's information is given by
h(x ⊔ (y,z)).  Alternatively, if Joan and Jonas share their information, then Eve's
information is given by h((x,y) ⊔ (x,z)).  Perhaps most interestingly, if Joan, Jonas
and Joanna share their information, then Eve's information is given by
h((x,y) ⊔ (x,z) ⊔ (y,z)).  Moreover, we know that Johnny has at least as much
information as Eve has in this situation,

    h(x,y,z) ≥ h((x,y) ⊔ (x,z) ⊔ (y,z)).

Thus, by comparing how much more information Johnny has relative to Eve in this situation, we can
define a new type of synergistic information content that quantifies how much information one gets
from knowing the full joint realisation to merely knowing all of the pairwise marginal realisations,

    h((x,y) ⊕ (x,z) ⊕ (y,z))
        = h(x,y,z) - h((x,y) ⊔ (x,z) ⊔ (y,z))
        = min(h(z|x,y),h(y|x,z),h(x|y,z)).



  Of course, these new ways to share joint information are not just restricted to the union
  information.  If Alice and Joanna share their information, then the information that Eve could
  have gotten from either is given by h(x ⊓ (y,z)).  It is also worthwhile noting
  that this quantity is not less than the information that Eve could have gotten from either Alice's
  information or Bob’s and Charlie’s information,
  
    h(x ⊓ (y,z)) ≥  h(x ⊓ (y ⊔ z)).

  Thus, we can also consider defining new types of synergistic information content associated with
  these this mixed type comparisons,
  
    h(x ⊓ (y ⊕ z)) = h(x⊓(y,z)) - h(x⊓(y ⊔ z)).

However, it is important to note that this quantity does not equal
min(h(x),min(h(z|y),h(y|z))).


With all of these new ways to share joint marginal information, it is not immediately clear how we
should decompose Johnny's information.  Nevertheless, let us begin by considering the algebraic
structure of joint information content.  From the inequality shannon_inequalities_pw, we know
that any pair of marginal information contents h(x) and h(y) are upper-bounded by the joint
information content h(x,y).  It is also easy to see that the joint information content is
idempotent, commutative and associative.  Together, these properties are sufficient for establishing
that the algebraic structure of joint information content is that of a join
semi-lattice <cit.> which we will denote by ⟨x; h( ,) ⟩.
semi-lattices shows the semi-lattices generated by n=2 and n=3 observers.

We now wish to establish the relationship between this semi-lattice of joint information content
⟨x; h( ,) ⟩ and the distributive lattice of shared marginal information
⟨x; h( ⊔ ), h( ⊓ ) ⟩.  In particular, since our aim is to decompose
Johnny's information, consider the relationship between the join semi-lattice
⟨x; h( ,) ⟩ and the meet semi-lattice ⟨x; h( ⊓ ) ⟩
which is also depicted in semi-lattices.  In contrast to the semi-lattice of union information
content ⟨x; h( ⊔ ) ⟩, the semi-lattice ⟨x; h( ,) ⟩ is
not connected to the semi-lattice ⟨x; h( ⊓ ) ⟩.  Although the intersection
information content absorbs the joint information content, since

    h(x ⊓(x, y)) = h(x)

for all h(x) and h(y), the joint information content does not absorb the intersection
information content since h(x, (x ⊓ y)) is equal to h(x,y) for h(x) ≥ h(y),
i.e. is not equal to h(x) as required for absorption.  Since the the join semi-lattice
⟨x; h( ,) ⟩ is not connected to the meet semi-lattice
⟨x; h( ⊓ ) ⟩ by absorption, their combined algebraic structure is not a
lattice.





Despite the fact that the overall algebraic structure is not a lattice, there is a lattice
sub-structure ⟨A(x), ≼⟩ within the general structure.  This
substructure is isomorphic to the redundancy lattice from the partial information
decomposition <cit.> (see also <cit.>), and its existence is a consequence
of the fact that the intersection information content absorbs the joint information content in
capWithJoint.  In order to identify this lattice, we must first determine the reduced set of
elements A(x) upon which it is defined.  We begin by considering the set of all possible
joint realisations which is given by P_1(x) where
P_1(x) = P(x) ∅.  Elements of this set P_1(x)
correspond to the elements from the join semi-lattice ⟨x; h( ,) ⟩, e.g. the
elements {x} and {x,y} correspond to h(x) and h(x,y), respectively.  In alignment with
<cit.>, we will call the elements of P_1(x) sources and denote them
by A_1,A_2,…,A_k.  Next, we consider set of all possible collections of
  sources which are given by the set P_1(P_1(x)).  Each collection of sources
corresponds to an element of the meet semi-lattice
⟨P_1(x) ; h( ⊓ ) ⟩, or a particular way in which we can evaluate the
intersection information content of a group of joint information contents.  For example, the
collections of sources {{x},{y}} and {{x},{y,z}} correspond to the h(x ⊓ y)
and h(x ⊓ (y,z))), respectively.  Not all of these collections of sources are
distinct, however.  Since the intersection information content absorbs the joint information
content, we can remove the element {{x},{x,y}} corresponding to h(x ⊓ (x,y))
as this information is already captured by the element {{x}} corresponding to h(x).  In
general, we can remove any collection of sources that corresponds to the intersection information
content between a source A_i and any source A_j that is in the down-set
↓A_i with respect to the join semi-lattice
⟨x; h( ,) ⟩.  (A definition of the down-set can be found in
<cit.>. Informally, the down-set ↓A is the set of all elements that
precede A.) By removing all such collections of sources, we get the following reduced set of
collections of sources,

    A(x) = {α∈P_1(P_1(x)) :
        ∀ A_i, A_j ∈α,  A_i ⊄A_j }.

Formally, this set corresponds to the set of antichains on the lattice
⟨P_1(x),⊆⟩, excluding the empty set <cit.>.

Now that we have determined the elements upon which the lattice sub-structure is defined, we must
show that they indeed form a lattice.  Recall that when constructing the set A(x), we
first considered the ordered elements of the semi-lattice
⟨x; h( ,) ⟩ and then subsequently consider the ordered elements of the
semi-lattice ⟨P_1(x) ; h( ⊓ ) ⟩.  Thus, we need to show
that these two orders can be combined together into one new ordering relation over the set
A(x).  This can be done by extending the approach underlying the construction of the set
A(x) to consider any pair of collections of sets α and β from
A(x).  In particular, the collection of sets β precedes the collection of sets
α if and only if for every source B from β, there exists a source A from
α such that A is in the down-set ↓B with respect to the
join-semi-lattice ⟨x; h( ,) ⟩, or formally,

    ∀ α, β∈A(x),
      (α≼β∀ B∈β,
        ∃ A∈α, A⊆B).

The fact that ⟨A(x),≼⟩ forms a lattice was proved by
<cit.> where the corresponding lattice is denoted
⟨A(X), ≼^'⟩ in their notation.  Furthermore, they showed that this
lattice is isomorphic to the distributive lattices, and hence the number of elements in the set
A(x) for n marginal observers is also given by the (n)-th Dedekind number
<cit.> (see also <cit.>).   also provided the meet
∧ and join ∨ operations for this lattice, which are given by

    α∧β   = α⊔β,  
    α∨β   = ↑α∩↑β,

where α denotes the set of minimal elements of α with respect to the
semi-lattice ⟨x; h( ,) ⟩.  (A definition of the set of minimal elements
can be found in <cit.>. Informally, α is the set of sources of
α that are not preceded by any other sources from α with respect to the semi-lattice
⟨x; h( ,) ⟩.)  This lattice ⟨A(x),≼⟩ is
the aforementioned sub-structure that is isomorphic to the redundancy lattice from
<cit.>.  However, as it is a lattice over information contents, it is actually
equivalent to the specificity lattice from <cit.>.  red_lattice depicts the
redundancy lattice of information contents for n=2 and n=3 marginal observers.




  Similar to how Eve's information is non-decreasing as we move up through the terms of the
  distributive lattice of shared information, the redundancy lattice of information contents enables
  to see that, for example, the information that Eve could have gotten from either Alice or Joanna
  h(x ⊓ (y,z)) is no less than the information that Eve could have gotten from Alice
  or Bob h(x ⊓ y).  Thus, by taking the information h(α) associated with the
  collection of sources α from A(x) and subtracting from it the information
  h(α_i) associated with any collection of sources α_i from the down-set
  ↓α, we can evaluate the unique information h(αα_i) provided
  by α relative to α_i.  Moreover, as per <cit.>, we can derive a
  function that quantifies the partial information content h_∂(α) associated
  with the collection of sources α that is not available in any of the collections of sources
  that are covered by α.  (The set of collections of sources that are covered by α is
  denoted α^-. A definition of the covering relation is provided in <cit.>.
  Informally, α^- is the set collections of sources that immediately precede α.)
  Formally, this function corresponds to the Möbius inverse of h on the redundancy lattice
  ⟨A(x), ≼⟩, and can be defined implicitly by

    h(α) = ∑_β≼α h_∂(β).

By subtracting away the partial information terms that strictly precede α from both sides, it
is easy to see that the partial information content h_∂(α) can be calculated
recursively from the bottom of the redundancy lattice of information contents,

    h_∂(α) = h(α) - ∑_β≺α h_∂(β),



As the following theorem shows, the partial information content h_∂(α) can be written
in closed-form.


  
  The partial information content h_∂(α) is given by
  
    h_∂(α)
             = h(α) - h(α^-_1 ⊔α^-_2 ⊔…⊔α^-_|α^-|) 
       = h(α) - max(h(α^-_1),h(α^-_2),…,h(α^-_|α^-|)),

  where each α^-_i is a collection of sets from α^-.


  For S⊆A(x), define the set-additive function
  
    f(S) = ∑_β∈S h_∂(β).

  From implicit, we have that h(α) = f(↓α).  The partial information
  can then by subtracting the set additive on the down-set ↓α from the set
  additive function on the strict down-set ↓̇α,
  
    h_∂(α) = f(↓α) - f(↓̇ α)
          = f(↓α) - f( ⋃_β∈α^-↓β),

  By applying the principle of inclusion-exclusion <cit.>, we get that
  
    h_∂(α) = f(↓α)
          - ∑_k=1^|α^-| (-1)^k-1∑_S⊆α^-
    
        |S|=k  f( ⋂_σ∈S↓σ).

  For any lattice L and A ⊆ L, we have that ⋂_a ∈ A↓ a is equal
  to ↓ ( ⋀ A ) <cit.>, and since the meet operation is given
  by the intersection information content, we have that
  
    h_∂(α)    = f(↓α)
          - ∑_k=1^|α^-| (-1)^k-1∑_S⊆α^-
    
          |S|=k  h(s_1 ⊓ s_2 ⊓…⊓ s_k) 
       = h(α) - h(α_1 ⊔α_2 ⊔…⊔α_|α^-|),

  where the final step has been made using pie_union and saf.


The closed-form solution closed_form from Theorem <ref> is the same as the
closed-form solution presented in Theorem A2 from <cit.>.  This, together with
the aforementioned fact that the lattice ⟨A(x),≼⟩ is equivalent to the
specificity lattice, means that each partial information content h_∂(α) is equal to
the partial specificity i^+_∂(α→ t) from (A22) of
<cit.>.  As such, the partial information decomposition present in this paper
is equivalent to the pointwise partial information decomposition presented in
<cit.>.

Let us now use the closed-form solution closed_form from Theorem <ref> to
evaluate the partial information contents for the n=2 redundancy lattice of information contents.
Starting from the bottom, we get the intersection information content,

    h_∂(x ⊓ y) = h(x ⊓ y),

followed by the unique information contents,

     h_∂(x)       = h(x)       - h(x ⊓y)       = h(x y), 

     h_∂(y)       = h(y)       - h(x ⊓y)       = h(y x),

and finally, the synergistic information content, 

    h_∂(x,y) = h(x,y) - h(x ⊔ y) = h(x ⊕ y).

It is clear that these partial information contents recover the intersection, unique and synergistic
information contents from Sections <ref> and <ref>.  Moreover, by inserting
these partial terms back into implicit for α={{x,y}}, we recover the earlier
decomposition bivar_decomp of Johnny's information,

    h(x,y) = h_∂(x ⊓ y) + h_∂(x) + h_∂(y) + h_∂(x,y)
      = h(x ⊓ y) + h(x  y) + h(y  x) + h(x ⊕ y).


Of course, our aim is to generalise this result such that we can decompose the joint information
content for an arbitrary number of marginal realisations.  This can be done by first evaluating the
partial information contents over the redundancy lattice corresponding to n marginal realisations,
and then subsequently inserting the results back into implicit for
α={{x_1, x_2, … ,x_n}}.  For example, we can invert the n=3 redundancy lattice of
information contents which yields the partial information contents shown in red_lattice.  (The
inversion is evaluated in the Appendix.)  When inserted back into implicit, we get the
following decomposition for Johnny's information,

    h(x,y,z)
           = h(x ⊓ y ⊓ z) 
          + h((x ⊓ y)  z) + h((x ⊓ z)  y)
          + h((y ⊓ z)  x) 
          + h(x ⊓ (y ⊕ z)) + h(y ⊓ (x ⊕ z))
          + h(z ⊓ (x ⊕ y)) 
          + h(x  (y,z)) + h(y  (x,z)) + h(z  (x,y))
        + h((x ⊕ y) ⊓ (x ⊕ z) ⊓ (y ⊕ z)) 
          + h((x ⊕ y) ⊓ (x ⊕ z)  (y,z))
          + h((x ⊕ y) ⊓ (y ⊕ z)  (x,z))
          + h((x ⊕ z) ⊓ (y ⊕ z)  (x,y)) 
          + h((x ⊕ y)  ((x,z) ⊔ (y,z)))
          + h((x ⊕ z)  ((x,y) ⊔ (y,z)))
          + h((y ⊕ z)  ((x,y) ⊔ (x,z))) 
          + h((x,y) ⊕ (x,z) ⊕ (y,z)).


Finally, we can also consider taking the expectation value of each term in the redundancy lattice of
information contents.  Since the expectation is a linear and monotonic operator, the resulting
expectation values will inherit the structure of the redundancy lattice of information contents and
so form a redundancy lattice of entropies, i.e. red_lattice with x, y, z and h
replaced by X, Y, Z and H respectively.  By inverting the n=2 redundancy lattice of
entropies, we can recover the decomposition bivar_decomp_avg from synergistic_entropy.
Furthermore, inverting the n=3 lattice generalises this result and is depicted in pid_3.






§ UNION AND INTERSECTION MUTUAL INFORMATION



Suppose that Alice, Bob and Johnny are now additionally and commonly observing the variable Z.
When a realisation (x,y,z) occurs, Alice's information given z is given by the conditional
information content h(x|z), while Bob's conditional information is given by h(y|z) and Johnny's
conditional information is given by h(x,y|z).  By using the same argument as in
marginal, it is easy to see that Eve's conditional information given z is given by the
conditional union information content,

    h(x ⊔ y | z) = max(h(x|z), h(y|z) ).

Likewise, we can define the conditional unique information contents and conditional intersection
information content, respectively,

     h(x y | z)       = h(x ⊔y | z) - h(y | z) = max(h(x|z) - h(y|z),0), 

     h(y x | z)       = h(x ⊔y | z) - h(x | z) = max(0,h(y|z) - h(x|z)), 

     h(x ⊓y|z)         = h(x|z) + h(y|z) - h(x ⊔y|z) = min(h(x|z), h(y|z)).

Furthermore, since Johnny's conditional information h(x,y|z) is no less than Eve's conditional
information content h(x ⊔ y | z), we can also define the conditional synergistic information
content,

    h(x ⊕ y|z) = h(x,y|z) - h(x ⊔ y | z) = min(h(y|x,z), h(x|y,z)).

Similar to bivar_decomp, we can decompose Johnny's conditional information h(x,y|z) into the
following components,

    h(x,y|z) = h(x ⊔ y|z) + h(x ⊕ y|z)
      = h(x ⊓ y|z) + h(x  y|z) + h(y  x|z) + h(x ⊕ y|z).

Moreover, similar to mi_diff, the conditional mutual information content is equal to the
difference between the conditional intersection information content and the conditional synergistic
information content,

    i(x;y|z) = h(x|z) + h(y|z) - h(x,y|z) = h(x ⊓ y|z) - h(x ⊕ y|z).

Notice that all of the above definitions directly correspond to the definitions of the unconditioned quantities, with all probability distributions conditioned on z here.















Let us now consider how much information each of our observers have about the commonly observed
realisation z.  The information that Alice has about z from observing x is given by the mutual
information content,

    i(x;z) = h(x) - h(x|z).

Similarly, Bob's information about z is given by i(y;z), while Johnny's information is given by
the joint mutual information content i(x,y;z).  Thus, the question naturally arises—are we able
to quantify how much information Eve has about the realisation z from knowing Alice's and Bob's
shared information?

Clearly, we could consider defining the union mutual information content,

    i(x ⊔ y ; z) = h(x ⊔ y) - h(x ⊔ y | z).

It is important to note that, while the mutual information can be defined in three different ways
i(x,z) = h(x) - h(x|z) = h(x) + h(z) - h(x,z) = h(z) - h(z|x), there is only one way in which one
can define this function.  (Indeed, this point aligns well with our argument based on exclusions
presented in <cit.>.)  Similar to union_mic, we could consider
respectively defining the unique mutual information contents, the intersection mutual information
content and synergistic mutual information content,

    i(x  y ; z)    = h(x  y) - h(x  y | z), 
    
      i(y  x ; z)    = h(y  x) - h(y  x | z), 
    
      i(x ⊓ y ; z)    = h(x ⊓ y) - h(x ⊓ y | z), 
     
      i(x ⊕ y ; z)    = h(x ⊕ y) - h(x ⊕ y | z).

Just like the mutual information content mutual_info_content, there is nothing to suggest that
these quantities are non-negative.  Of course, the mutual information or expected mutual information
content mutual_info is non-negative.  Thus, with this in mind, consider defining the union
mutual information

    I(X ⊔ Y ; Z) = E_XYZ[i(x ⊔ y;z)].

However, there is nothing to suggest that this function is non-negative.  Consequently, it is
dubious to claim that this function represents Eve's expected information about Z, and is
similarly fallacious to say that Eve's information about z is given by the union mutual
information content union_mic.  Indeed, by inserting the definitions union_info_content
and cond_union_info_content into union_mic, it is easy to see why it is difficult to
interpret these functions,

    i(x ⊔ y ; z)    = max(h(x),  h(y)) - max(h(x|z),  h(y|z)) 
       = max(min(i(x;z), h(x)-h(y|z)), min(h(y)-h(x|z), i(y;z))).

That is, the union mutual information content can mix the information content provided by one
realisation with the conditional information content provided by another.  Thus, there is no
guarantee that this function's expected value will be non-negative.  It is perhaps best to interpret
this function as being a difference between two surprisals, rather than a function which represent
information.  Of course, similar to the multivariate mutual information mmi, the union
mutual information can be used a summary quantity provided one is careful not to misinterpret its
meaning.  The same is true for the unique mutual informations,
intersection mutual information and synergistic mutual information, which we can similarly define,

    I(X  Y ; Z)    = E_XYZ[i(x  y;z)], 
    
      I(Y  X ; Z)    = E_XYZ[i(y  x;z)], 
    
      I(X ⊓ Y ; Z)    = E_XYZ[i(x ⊓ y;z)], 
    
      I(X ⊕ Y ; Z)    = E_XYZ[i(x ⊕ y;z)].


Despite lacking the clear interpretation that we had for the information contents, these functions
share a similar algebraic structure.  For example, by using bivar_decomp and
bivar_decomp_cond, we can decompose the mutual information content into the following
components,

    i(x,y;z) =  i(x ⊓ y ; z) + i(x  y ; z) + i(y  x ; z) +  i(x ⊕ y ; z),

which is similar to the earlier decomposition of the joint entropy bivar_decomp.  Moreover,
similar to mi_diff, by using mi_diff and mi_diff_cond, we get that the multivariate
mutual information content is given by the difference between the intersection mutual information
content and the synergistic mutual information content,

    i(x;y;z)    = i(x;y) - i(x;y|z)
                 = h(x ⊓ y) - h(x ⊕ y) - h(x ⊓ y) + h(x ⊕ y|z) 
       = i(x ⊓ y; z) - i(x ⊕ y; z).

Of course, since the expectation value is a linear operator, both of these results can be carried
over to the joint mutual information.  Hence, the mutual information can be decomposed into the
following components,

    I(X,Y;Z) =  I(X ⊓ Y ; Z) + I(X  Y ; Z) + I(Y  X ; Z) +  I(X ⊕ Y ; Z).

while the the multivariate mutual information is equal to the intersection mutual information minus
the synergistic mutual information,

    I(X;Y;Z)    = I(X;Y) - I(X;Y|Z)
                 = H(X ⊓ Y) - H(X ⊕ Y) - H(X ⊓ Y) + H(X ⊕ Y|Z) 
       = I(X ⊓ Y; Z) - I(X ⊕ Y; Z).

This latter result aligns Williams and Beer's prior result that the multivariate mutual information
conflates redundant and synergistic information <cit.>.






























§ CONCLUSION



The main aim of this paper has been to understand and quantify the distinct ways that a set of
marginal observers can share their information with some non-observing third party.  To accomplish
objective, we examined the distinct ways in which two marginal observers, Alice and Bob, can share
their information with the non-observing individual, Eve, and introduced several novel measures of
information content: the union, intersection and unique information contents.  We then investigated
the algebraic structure of these new measures of shared marginal information and showed that the
structure of shared marginal information is that of a distributive lattice.  Furthermore, by using
the fundamental theorem of distributive lattices, we showed that these new measures are isomorphic
to the set union and intersections.  This isomorphism is similar to Yeung's correspondence between
multivariate mutual information and signed measure <cit.>.  However, in
contrast to Yeung's correspondence, the measures of information content presented in this paper are
non-negative.  Moreover, these measures maintain a clear operational meaning regardless of the
number of realisations or variables involved.  (This is, of course, excepting the mutual information
contents presented in info which are not non-negative; whether or not one should use such
quantities is open to debate.)

The secondary objective of this paper has been to understand and demonstrate how we can use the
measures of shared information content to decompose multivariate information.  We began by comparing
the union information content to the joint information content and used this comparison to define a
measure of synergistic information content.  We showed how one can use this measure, together with
the measures of shared information content, to decompose the joint information content.  Next, we
compared the algebraic structure of joint information to the lattice structure of shared
information, and showed how one can find the redundancy lattice from the partial information
decomposition <cit.> embedded within this larger algebraic structure.  More
specifically, since this paper considers information contents, this redundancy lattice is actually
same as the specificity lattice from pointwise partial information decomposition
<cit.>.

This result links the work presented in this paper to the existing body of theoretical literature on
information decomposition <cit.>, and its applications <cit.>.  (For a
brief summary of this literature, see <cit.>.)  In contrast to the pointwise partial
information decomposition <cit.>, most of these approaches
aim to decompose the average mutual information rather than the information content.  One exception
to this statement is due to <cit.>, who proposes a method of information decomposition
based upon the entropy.  Moreover, Ince discusses how this decomposition can be applied to the
information content (or in Ince's terminology, the local entropy).  Of particular relevance to this
paper, Ince obtains a result that is equivalent to mi_diff whereby the mutual information
content is equal to the redundant information content minus the synergistic information content
<cit.>.  However, Ince's definition of redundant information content differs from
that of the intersection information content in mi_diff.  To be specific, it is based upon the
sign of the multivariate mutual information content (or pointwise co-information), which is
interpreted as a measure of “the set-theoretic overlap” of multiple information contents (or local
entropies) <cit.>.  However, as discussed in intro, this set-theoretic
interpretation of the multivariate mutual information (co-information) is problematic.  To account
for these difficulties, Ince disregards the negative values, defining the redundant information
content to equal to the multivariate mutual information when it is positive, and to be zero
otherwise.












There are a number of avenues of inquiry for which this research will yield new insights,
particularly in complex systems, neuroscience, and communications theory.  For instance, these
measures might be used to better understand and quantify distributed intrinsic computation
<cit.>.  It is well known that that dynamics of individual regions
in the brain depend synergistically on multiple other regions; synergistic information content might
provide a means to quantify such dependencies in neural data <cit.>.  Furthermore, these measures
might be helpful for quantifying the synergistic encodings used in network coding <cit.>.
Finally, it is well-known that many biological traits are not dependent on any one gene, but rather
are synergistically dependent on two or more genes, and the decomposed information provides a means
to quantify the unique, redundant and synergistic dependencies between a trait and a set of
genes <cit.>.



 



 §.§ Acknowledgements

The authors would like to thank Nathan Harding, Richard Spinney, Daniel Polani, Leonardo Novelli and
Oliver Cliff for helpful discussions relating to this manuscript.





§ APPENDIX


We can use the closed-form closed_form to evaluate the partial information contents for the
n=3 redundancy lattice of information contents.  Starting from the bottom and working up through
the redundancy lattice, we have the following partial information contents:

    h_∂(x ⊓ y ⊓ z)
           = h(x ⊓ y ⊓ z); [0]
    
      h_∂(x ⊓ y)
           = h(x ⊓ y) - h(x ⊓ y ⊓ z) 
       = h((x ⊓ y)  z); [0]
    
      h_∂(x ⊓ (y,z))
           = h(x ⊓ (y,z)) - h((x ⊓ y) ⊔ (x ⊓ z))
       = h(x ⊓ (y,z)) - h(x ⊓ (y ⊔ z)) 
       = h(x ⊓ (y ⊕ z)); [0]
    
      h_∂(x)
           = h(x) - h(x ⊓ (y,z))
       = h(x  (y,z)); [0]
    
      h_∂((x,y)⊓(x,z)⊓(y,z))
           = h((x,y)⊓(x,z)⊓(y,z))
          - h((x⊓(y,z))⊔(y⊓(x,z))⊔(z⊓(x,y))) 
       = h((x,y)⊓(x,z)⊓(y,z)) - h(x ⊓ (y ⊕ z))
          - h(y ⊓ (x ⊕ z)) - h(z ⊓ (x ⊕ y)) 
            - h((x ⊓ y)  z ) - h((x ⊓ z)  y )
          - h((y ⊓ z)  x ) - h(x ⊓ y ⊓ z) 
       = h((x,y)⊓(x,z)⊓(y,z)) - h(x ⊓ (y, z))
          - h(y ⊓ (x ⊕ z)) - h(z ⊓ (x ⊕ y)) 
            - h((y ⊓ z)  x ) By L9.
       = h((x ⊕ y) ⊓ (x ⊕ z) ⊓ (y ⊕ z)); [0]
    
      h_∂((x,y) ⊓ (x,z) )
           = h((x,y) ⊓ (x,z))
          - h(x ⊔ ((x,y) ⊓ (x,z) ⊓(y,z))) 
       = h((x,y) ⊓ (x,z)) -h(x) - h((x,y) ⊓ (x,z) ⊓(y,z)) 
            + h(x ⊓ ((x,y) ⊓ (x,z) ⊓(y,z))) 
       = h(((x,y)⊓(x,z))  (y,z)) - h(x  (y,z))
          By Lemma <ref>.
       = h( ((x,y)  (y,z)) ⊓ ((x,z)  (y,z)) ) - h(x  (y,z))
          By Lemma <ref>.
       =h((x ⊕ y) ⊓ (x ⊕ z)  (y,z)); [0]
    
      h_∂(x,y)
           = h(x,y) - h(((x,y)⊓(x,z)) ⊔ ((x,y)⊓(y,z))) 
       = h(x,y) - h((x,y) ⊓ ((x,z) ⊔ (y,z))) 
       = h((x,y)  ((x,z) ⊔ (y,z))) By Lemma <ref>.
    
           = h((x ⊕ y)  ((x,z) ⊔ (y,z))); [0]
    
      h_∂(x,y,z)
           = h(x,y,z) - h((x,y) ⊔ (x,z) ⊔ (y,z)) 
       = h((x,y) ⊕ (x,z) ⊕ (y,z)).
 

 We have the following identity,
  
  
    h((x ⊓ y)  z) = h((x  z) ⊓ (y  z)).




  From uni_ic_x and max_min_two, we have that
  
    h((x ⊓ y)  z) 
             = max(min(h(x),h(y))-h(z), 0) = max(min(h(x)-h(z), h(y)-h(z)), 0) 
       = min(max(h(x)-h(z),0), max(h(y)-h(z),0)) = h((x  z)⊓(y  z)),

  where we have used the fact that max(a, min(b,c)) is equal to min(max(a,b), max(a,c)).




  We have the following identity,
  
    h(((x,y)  (y,z)) ⊓ ((x,z)  (y,z)) )
          = h(x  (y,z)) + h((x ⊕ y) ⊓ (x ⊕ z)  (y,z)).




  From bivar_decomp, we have that
  
    h((x,y)  (y,z))
             = h((x ⊓ y)  (y,z)) + h((x  y)  (y,z)) + h((y  x)  (y,z)) + h((x ⊕ y)  (y,z)) 
       = h(x  (y,z)) + h((y  x)  (y,z))
            + h((x ⊕ y)  (y,z)),

  and since
  
    h((y  x)  (y,z)) = max(h(y  x)-h(y,z),0) = 0,

  we get that
  
    h((x,y)  (y,z)) = h(x  (y,z)) + h((x ⊕ y)  (y,z)).

  Finally, by inserting this result into max_min_two, we get that
  
    h(((x,y)  (y,z)) ⊓ ((x,z)  (y,z)) 
       = min(h(x  (y,z)) + h((x ⊕ y)  (y,z)),  
            h(x  (y,z)) + h((x ⊕ z)  (y,z))) 
       = h(x  (y,z))
            + h(((x ⊕ y)  (y,z)) ⊓ ((x ⊕ z)  (y,z))).





  
  We have the following identity, 
  
    h(((x,y)  ((x,z) ⊔ (y,z)) ) = h(((x ⊕ y)  ((x,z) ⊔ (y,z)) ).




  By bivar_decomp, we have that
  
    h((x,y)  ((x.z) ⊔ (y,z)))
             = h((x ⊓ y)  ((x,z) ⊔ (y,z)))
            + h((x  y)  ((x,z) ⊔ (y,z))) 
          + h((y  x)  ((x,z) ⊔ (y,z)))
            + h((x ⊕ y)  ((x,z) ⊔ (y,z))).

  Since h(x ⊓ y), h(x  y) and h(y  x) are less than h(((x,z) ⊔ (y,z)),
  from uni_ic_x we have that h((x ⊓ y)  ((x,z) ⊔ (y,z))),
  h((x  y)  ((x,z) ⊔ (y,z))) and
  h((y  x)  ((x,z) ⊔ (y,z))) are all equal to 0.



  
  We have the following identity, 
  
    h(x ⊓ (y  x)) = 0.




  We have that
  
    h(x ⊓ (y  x)) = h(x ⊓ (x,y)) - h(x ⊓ x)
        = h(x ⊓ x) - h(x) = 0,

  where we have used intersectionIdempotent and capWithJoint.



  
  We have the following identity, 
  
    h((y  x) ⊓ (y, z)) = h(y  x ).




  We have that
  
    h((y  x) ⊓ (y, z))
             = h((y  x) ⊓ (y  x, y ⊓ x, z)).

  Using capWithJoint, this then reduces via to
  
    h((y  x) ⊓ (y, z)) = h(y  x ).




  
  We have the following identity, 
  
    h(x ⊓ (x ⊕ z)) = 0.




  We have
  
    h(x ⊓ (x ⊕ z))
             = h(x ⊓ (x, z)) - h(x ⊓ x) + h(x ⊓ (z  x)) ,

  which then reduces via capWithJoint, intersectionIdempotent and L4 to:
  
    h(x ⊓ (x ⊕ z))
             = h(x) - h(x) + h(x ⊓ (z  x)) + 0
       =0.




  
  We have the following identity, 
  
    h((y  x) ⊓ (x ⊕ z)) = h(y ⊓ (x ⊕ z) ).




  We have that
 
    h(y ⊓ (x ⊕ z) )
             = h((y  x) ⊓ (x ⊕ z))
            + h((y ⊓ x) ⊓ (x ⊕ z)) .

  But since h(x ⊓ (x ⊕ z)) = 0 via L6 and therefore h((y ⊓ x) ⊓ (x ⊕ z)) = 0, then we have 
  
    h(y ⊓ (x ⊕ z) ) = h((y  x) ⊓ (x ⊕ z))

  as required.



  
  We have the following identity, 
  
    h((x ⊕ y) ⊓ (x, z) ⊓ (y,z) ) = h(z ⊓ (x ⊕ y) ) + h((x ⊕ y) ⊓ (x ⊕ z) ⊓ (y ⊕ z) ).




  We have that
  
    h((x ⊕ y) ⊓ (x, z) ⊓ (y,z) )
             = h((x ⊕ y) ⊓ x ⊓ (y,z) )
            + h((x ⊕ y) ⊓ (z  x) ⊓ (y,z) ) 
            + h((x ⊕ y) ⊓ (x ⊕ z) ⊓ (y,z) ).

  Then, by using L6, we get that
  
    h((x ⊕ y) ⊓ (x, z) ⊓ (y,z) )
             = h((x ⊕ y) ⊓ (z  x) ⊓ (y,z) )
            + h((x ⊕ y) ⊓ (x ⊕ z) ⊓ (y,z) ) 
       = h((x ⊕ y) ⊓ (z  x) ⊓ y )
            + h((x ⊕ y) ⊓ (z  x) ⊓ (z  y) ) 
            + h((x ⊕ y) ⊓ (z  x) ⊓ (y ⊕ z) )
            + h((x ⊕ y) ⊓ (x ⊕ z) ⊓ y ) 
            + h((x ⊕ y) ⊓ (x ⊕ z) ⊓ (z  y) )
            + h((x ⊕ y) ⊓ (x ⊕ z) ⊓ (y ⊕ z) ) 
       = h((x ⊕ y) ⊓ (z  x) ⊓ (z  y)
            + h((x ⊕ y) ⊓ (x ⊕ z) ⊓ (y ⊕ z) ),

  where we have used L6 four more times.  Finally, using L7, we get that
  
    h((x ⊕ y) ⊓ (x, z) ⊓ (y,z) )
             = h((x ⊕ y) ⊓ z )
            + h((x ⊕ y) ⊓ (x ⊕ z) ⊓ (y ⊕ z) ),

  as required.



  
  We have the following identity, 
  
    h((x, y) ⊓ (x, z) ⊓ (y,z) )
             = h(x ⊓ (y,z) ) + h(y ⊓ (x ⊕ z) )
            + h(z ⊓ (x ⊕ y) ) + h((y ⊓ z)  x ) 
            + h((x ⊕ y) ⊓ (x ⊕ z) ⊓ (y ⊕ z) ).




  We have that
  
    h((x, y) ⊓ (x, z) ⊓ (y,z) )
             = h(x ⊓ (x, z) ⊓ (y,z) )
            + h((y  x) ⊓ (x, z) ⊓ (y,z) ) 
            + h((x ⊕ y) ⊓ (x, z) ⊓ (y,z) ) 
       = h(x ⊓ (y,z) )
            + h((y  x) ⊓ (x, z) ) 
            + h((x ⊕ y) ⊓ (x, z) ⊓ (y,z) ),

  where we have used capWithJoint and L5.  Next, we have that
  
    h((x, y) ⊓ (x, z) ⊓ (y,z) )
             = h(x ⊓ (y,z) )
            + h((y  x) ⊓ x ) 
            + h((y  x) ⊓ (z  x) ) 
            + h((y  x) ⊓ (x ⊕ z) ) 
            + h((x ⊕ y) ⊓ (x, z) ⊓ (y,z) ) 
       = h(x ⊓ (y,z) )
            + 0 
            + h((y ⊓ z)  x ) 
            + h(y ⊓ (x ⊕ z) ) 
            + h((x ⊕ y) ⊓ (x, z) ⊓ (y,z) )

  where we have used L1, L4 and L7.  Finally, we have that
  
    h((x, y) ⊓ (x, z) ⊓ (y,z) )
             = h(x ⊓ (y,z) )
            + h((y ⊓ z)  x ) 
            + h(y ⊓ (x ⊕ z) ) 
            + h(z ⊓ (x ⊕ y) ) 
            + h((x ⊕ y) ⊓ (x ⊕ z) ⊓ (y ⊕ z) ),

  where we have used via L8 to get the required result.


