On extractable shared information
Johannes Rauh Pradeep Kr. Banerjee Eckehard Olbrich Jürgen Jost Nils Bertschinger

Private Information Retrieval
 
from MDS Coded Data with Colluding Servers: 

Settling a Conjecture by Freij-Hollanti et al.


     Hua Sun and Syed A. Jafar 





    
==============================================================================================================================






 We consider the problem of quantifying the information shared by a pair of random variables X_1,X_2 about another variable S. We propose a new measure of shared information, called extractable shared information, that is left monotonic; that is, the information shared about S is bounded from below by the information shared about f(S) for any function f.
  We show that our measure leads to a new nonnegative decomposition of the mutual information I(S;X_1X_2) into shared, complementary and unique components.
  We study properties of this decomposition and show that a left monotonic shared information is not compatible with a Blackwell interpretation of unique information.
  We also discuss whether it is possible to have a decomposition in which both shared and unique information are left monotonic.





Keywords: Information decomposition; multivariate mutual information; left monotonicity; Blackwell order












§ INTRODUCTION


A series of recent papers have focused on the bivariate information decomposition problem <cit.>.  Consider three random variables S, X_1, X_2 with finite alphabets ,  and , respectively.  The total information that the pair (X_1,X_2) convey about the target S can have aspects of shared or redundant information (conveyed by both X_1 and X_2), of unique information (conveyed exclusively by either X_1 or X_2), and of complementary or synergistic information (retrievable only from the the joint variable (X_1,X_2)). In general, all three kinds of information may be present concurrently. One would like to express this by decomposing  the mutual information I(S;X_1X_2) into a sum of nonnegative components with a well-defined operational interpretation. 

One possible application area is in the neurosciences. 
In <cit.>, it is argued that such a decomposition can provide a framework to analyze neural information processing using information theory that can integrate and go beyond previous attempts.

For the general case of k finite source variables (X_1,…,X_k), Williams and Beer <cit.> proposed the partial information lattice framework that specifies how the total information about the target S is shared across the singleton sources and their disjoint or overlapping coalitions.
The lattice is a consequence of certain natural properties of shared information (sometimes called the Williams–Beer axioms).
In the bivariate case (k=2), the decomposition has the form

    I(S;X_1X_2)    = (S;X_1,X_2)_shared + (S; X_1,X_2)_complementary + (S;X_1\ X_2)_unique (X_1 wrt X_2) + (S;X_2\ X_1)_unique (X_2 wrt X_1), 
    
    I(S;X_1)    = (S;X_1,X_2) + (S;X_1\ X_2)  ,
    
    I(S;X_2)    = (S;X_1,X_2) + (S;X_2\ X_1)  ,

where (S;X_1,X_2), (S;X_1\ X_2), (S;X_2\ X_1), and (S;X_1,X_2) are nonnegative functions that depend continuously on the joint distribution of (S,X_1,X_2). 
The difference between shared and complementary information is the familiar co-information <cit.> (or interaction information <cit.>), a symmetric generalization of the mutual information for three variables, 

    CoI(S;X_1,X_2)   =I(S;X_1)-I(S;X_1|X_2)=(S;X_1,X_2)-(S; X_1,X_2).

Equations (<ref>) to (<ref>) leave only a single degree of freedom, i.e., it suffices to specify either a measure for , for  or for .  

Williams and Beer not only introduced the general partial information framework, but also proposed a measure of SI to
fill this framework.  While their measure has subsequently been criticized for “not measuring the right thing”
<cit.>,
there has been no successful attempt to find better measures, except for the bivariate case
(k=2) <cit.>.  One problem seems to
be the lack of a clear consensus on what an ideal measure of shared (or unique or complementary) information should look
like and what properties it should satisfy. In particular, the Williams–Beer axioms only put crude bounds on the values
of the functions ,  and .  Therefore, additional axioms have been proposed by various
authors <cit.>.
Unfortunately, some of these properties contradict each other <cit.>, and the question for
the right axiomatic characterization is still open.

The Williams–Beer axioms do not say anything about what should happen when the target variable S undergoes a local transformation. 
In this context, the following left monotonicity property was proposed in <cit.>:

	
  (LM) (S;X_1,X_2) ≥(f(S);X_1,X_2) for any function f.
	asdf (left monotonicity)
	

Left monotonicity for unique or complementary information can be defined similarly.
The property captures the intuition that shared information should only decrease if the target performs some local operation (e.g., coarse graining) on her variable S.  
As argued in <cit.>, left monotonicity of shared and unique information are indeed desirable properties. Unfortunately, none of the measures of shared information proposed so far satisfy left monotonicity. 

In this contribution, we study a construction that enforces left monotonicity.  Namely, given a measure of shared information , define

    (S;X_1,X_2) := sup_f:→' (f(S);X_1,X_2),

where the supremum runs over all functions f:→' from the domain of S to an arbitrary finite set '.
By construction,  satisfies left monotonicity, and  is the smallest function bounded from below by  that satisfies left monotonicity.

Changing the definition of shared information in the information decomposition
framework Equations (<ref>)–(<ref>) leads to new definitions of unique and complementary information:

    (S;X_1\ X_2)     := I(S;X_1) - (S;X_1,X_2), 
    (S;X_2\ X_1)     := I(S;X_2) - (S;X_1,X_2), 
    (S;X_1,X_2)     := I(S;X_1X_2) - (S;X_1,X_2)- (S;X_1\ X_2) - (S;X_2\ X_1).

In general, (S;X_1∖ X_2)≠(S;X_1∖ X_2) :=
sup_f:→' (f(S);X_1∖ X_2).  Thus, our construction cannot enforce left monotonicity for
both UI and SI in parallel.


Lemma <ref> shows that ,  and  are nonnegative and thus define a
nonnegative bivariate decomposition.
We study this decomposition in Section <ref>.  In Theorem <ref>, we show that our construction is not compatible with a decision-theoretic interpretation of unique information proposed in <cit.>.  In Section <ref>, we ask whether it is possible to find an information decomposition in which both shared and unique information measures are left monotonic. Our construction cannot directly be generalized to ensure left monotonicity of two functions simultaneously. Nevertheless, it is possible that such a decomposition exists, and in Proposition <ref>, we prove bounds on the corresponding shared information measure.

Our original motivation for the definition of  was to find a bivariate decomposition in which the shared information satisfies left monotonicity. However, one could also ask whether left monotonicity is a required property of shared information, as put forward in <cit.>.
In contrast, <cit.> argue that redundancy can also arise by means of a mechanism.
Applying a function to S corresponds to such a mechanism that singles out a certain aspect from S.  Even if all the X_i share nothing about the whole S, they might still share information about this aspect of S, which means that the shared information will increase. With this intuition, we can interpret  not as an improved measure of shared information, but as a measure of extractable shared information, because it asks for the maximal amount of shared information that can be extracted from S by further processing S by a local mechanism.  More generally, one can apply a similar construction to arbitrary information measures. We explore this idea in Section <ref> and discuss probabilistic generalizations and relations to other information measures.
In Section <ref>, we apply our construction to existing measures of shared information.





§ PROPERTIES OF INFORMATION DECOMPOSITIONS





 §.§ The Williams–Beer Axioms


Although we are mostly concerned with the case k=2, let us first recall the three axioms that Williams and
Beer <cit.> proposed for a measure of shared information for arbitrarily many arguments:

	
  (S) (S;X_1,…,X_k) is symmetric under permutations of X_1,…,X_k,   (Symmetry)
	
  (SR) (S;X_1) = I(S;X_1), (Self-redundancy)
	
  (M) (S;X_1,…,X_k-1,X_k) ≤(S;X_1,…,X_k-1),
 with equality if
	X_i=f(X_k) for some i<k and some function f.  (Monotonicity)

Any measure of  satisfying these axioms is nonnegative.

Moreover, the axioms imply the following:

	
  (RM) (S;X_1,…,X_k) ≥(S;f_1(X_1),…,f_k(X_k)) for all
	functions f_1,…,f_k. (right monotonicity)


Williams and Beer also defined a function

    I_min(S;X_1,…,X_k) = ∑_sP_S(s)min_i{∑_x_iP_X_i|S(x_i|s)logP_S|X_i(s|x_i)/P_S(s)}

and showed that I_min satisfies their axioms.



 §.§ The Copy example and the Identity Axiom


Let X_1,X_2 be independent uniformly distributed binary random variables, and consider the copy function
Copy(X_1,X_2):=(X_1,X_2). One point of criticism of I_min is the fact that
X_1 and X_2 share I_min(Copy(X_1,X_2);X_1,X_2) = 1bit about
Copy(X_1,X_2) according to I_min, even though they are independent. 
<cit.> argue that the shared information about the copied pair should equal the mutual information:

	
  (Id) (Copy(X_1,X_2);X_1,X_2)=I(X_1;X_2).  (Identity)


Ref. <cit.> also proposed a bivariate 
measure of shared information that satisfies
(Id).  Similarly, the measures of bivariate shared information proposed in
<cit.> satisfies (Id).
However, (Id) is incompatible with a nonnegative information decomposition according to the Williams–Beer axioms for k≥ 3 <cit.>.

On the other hand, Ref. <cit.> uses an example from game theory to give an intuitive explanation how even independent variables X_1 and X_2 can have nontrivial shared information.  However, in any case the value of 1bit assigned by I_min is deemed to be too large.





 §.§ The Blackwell property and property (∗)


One of the reasons that it is so difficult to find good definitions of shared, unique or synergistic information is that a clear operational idea behind these notions is missing.  Starting from an operational idea about decision problems,
Ref. <cit.> proposed the following property for the unique information, which we now
propose to call Blackwell property:

	
  (BP) For a given joint distribution P_SX_1X_2, (S;X_1\ X_2) vanishes if and only if there exists a random
	variable X_1' such that S-X_2-X_1' is a Markov chain and P_SX_1'=P_SX_1.
	

In other words, the channel S → X_1 is a garbling or degradation of the channel S → X_2.  Blackwell's
theorem <cit.> implies that this garbling property is equivalent to the fact that any decision
problem in which the task is to predict S can be solved just as well with the knowledge of X_2 as with the
knowledge of X_1.  We refer to Section 2 in <cit.> for the details.

Ref. <cit.> also proposed the following property:

	
  (*)  and  depend only on the marginal distributions P_SX_1 and P_SX_2 of the
	pairs (S,X_1) and (S,X_2).

This property was in part motivated by (BP), which also depends only on the channels S→ X_1 and S→ X_2 and thus on P_SX_1 and P_SX_2.  Most information decompositions proposed so far satisfy property (*).



§ EXTRACTABLE INFORMATION MEASURES



One can interpret SI as a measure of extractable shared information. We explain this idea in a more general setting.

For fixed k, let IM(S;X_1,…,X_k) be an arbitrary information measure that measures one aspect of the
information that X_1,…,X_k contain about S.  At this point, we do not specify what precisely an
information measure is, except that it is a function that assigns a real number to any joint distributions of
S,X_1,…,X_k.  The notation is, of course, suggestive of the fact that we mostly think about one of the measures ,  or , in which the first argument plays a special role.  However, IM could also be the mutual information I(S;X_1),
the entropy H(S), or the coinformation CoI(S;X_1,X_2).  
We define the corresponding extractable information measure as

    IM(S;X_1,…,X_k) := sup_f IM(f(S);X_1,…,X_k),

where the supremum runs over all functions f:↦' from the domain of S to an arbitrary finite set '. The intuition is that IM is the maximal possible amount of IM one can “extract” from (X_1,…,X_k) by transforming S. Clearly, the precise interpretation depends on the interpretation of IM.

This construction has the following general properties:

	
  * Most information measures satisfy  IM(O;X_1,…,X_k)=0 when O is a constant random variable.  Thus, in this case, IM(S;X_1,…,X_k)≥ 0. Thus, for example, even though the coinformation can be negative, the extractable coinformation is never negative.
	
  * Suppose that IM satisfies left monotonicity.
	Then, IM = IM. For example, entropy H and mutual information I satisfy left monotonicity, and so H=H
	and I=I. Similarly, as shown in <cit.>, the measure of unique
	information  defined in <cit.> satisfies left monotonicity, and so 
	=.
	
  * In fact, IM is the smallest left monotonic information measure that is at least as large as IM.


The next result shows that our construction preserves monotonicity properties of the other arguments of IM.
It follows that, by iterating this construction, one can construct an information measure that is monotonic in all arguments.

	
	Let f_1,…,f_k be fixed functions.  If IM satisfies IM(S;f_1(X_1),…,f_k(X_k))≤
	IM(S;X_1,…,X_k) for all S, then IM(S;f_1(X_1),…,f_k(X_k))≤IM(S;X_1,…,X_k) for all S.


	Let f^* = _f{IM (f(S);f_1(X_1),…,f_k(X_k))}. Then,
	
    IM(S;f_1(X_1),…,f_k(X_k))
    	= IM(f^*(S);f_1(X_1),…,f_k(X_k))
    
    	≤^(a) IM(f^*(S);X_1,…,X_k)≤sup_f IM(f(S);X_1,…,X_k)
    	
    	=IM(S;X_1,…,X_k),
 
	where (a) follows from the assumptions.



As a generalization to the construction, instead of looking at “deterministic extractability,” one can also look at
“probabilistic extractability” and replace f by a stochastic matrix.  This leads to the definition

    IM(S;X_1,…,X_k) := sup_P_S'|S IM(S';X_1,…,X_k),

where the supremum now runs over all random variables S' that are independent of X_1,…,X_k given S.  The
function IM is the smallest function bounded from below by IM that satisfies

	
  (PLM) IM(S;X_1,X_2) ≥IM(S';X_1,X_2) whenever S' is independent of X_1,X_2 given S.
	
	


An example of this construction is the intrinsic conditional information

I(X;Y↓ Z) := min_P_Z'|Z I(X;Y|Z'),

which was defined in <cit.> to study the secret-key rate, which is the maximal rate at which a secret can be generated by two agents knowing X or Y, respectively, such that a third agent who knows Z has arbitrarily small information about this key.  The min instead of the max in the definition implies that I(X;Y↓ Z) is “anti-monotone” in Z.

In this paper, we restrict ourselves to the deterministic notions, since many of the
properties we want to discuss can already be explained using deterministic extractability.
Moreover, the optimization problem (<ref>) is a finite optimization problem and thus much easier to solve
than Equation (<ref>).



§ EXTRACTABLE SHARED INFORMATION



We now specialize to the case of shared information. The first result is that when we apply our construction to a measure of shared information that belongs to a bivariate information decomposition, we again obtain a bivariate information decomposition.

	
	Suppose that  is a measure of shared information, coming from a nonnegative bivariate information decomposition (satisfying Equations (<ref>) to (<ref>)).  Then,  defines a nonnegative information decomposition; that is, the derived functions
	
    (S;X_1\ X_2)     := I(S;X_1) - (S;X_1,X_2), 
    (S;X_2\ X_1)     := I(S;X_2) - (S;X_1,X_2), 
    and   (S;X_1,X_2)     := I(S;X_1X_2) - (S;X_1,X_2) - (S;X_1\ X_2) - (S;X_2\ X_1)

	are nonnegative. These quantities relate to the original decomposition by
	
    a)     (S;X_1,X_2) ≥(S;X_1,X_2),
    
    	b)     (S;X_1,X_2) ≥(S;X_1,X_2),
    
    	c)     (f^*(S);X_1\ X_2) ≤(S;X_1\ X_2) 
    	
    	≤(S;X_1\ X_2),

	where f^* is a function that achieves the supremum in Equation (<ref>).


	
	
    a)   (S;X_1,X_2) ≥(S;X_1,X_2) ≥ 0,
    
    	b)   (S;X_1,X_2)=(S;X_1,X_2)-CoI(S;X_1,X_2)
    	
    	≥(S;X_1,X_2) - CoI(S;X_1,X_2)
       ≥(S;X_1,X_2) ≥ 0, 
    
    	c)   (S;X_1\ X_2)=I(S;X_1)-(S;X_1,X_2)
    	
    	≤ I(S;X_1) - (S;X_1,X_2)=(S;X_1\ X_2),
    
    	   (S;X_1\ X_2)= I(S;X_1)-(S;X_1,X_2)
    	
    	≥ I(f^*(S);X_1)-(f^*(S);X_1,X_2)
        = (f^*(S);X_1\ X_2) ≥ 0,

	where we have used the data processing inequality.


	
	
		
  * If  satisfies (*), then  also satisfies (*).
		
  * If  is right monotonic, then  is also right monotonic.
	


	(1) is direct, and (2) follows from Lemma <ref>.


Without further assumptions on , we cannot say much about when  vanishes.  However, the condition that  vanishes has strong consequences.

	
	Suppose that (S;X_1\ X_2) vanishes, and let f^* be a function that achieves the supremum in Equation (<ref>).  Then, there is a Markov chain .  Moreover, (f^*(S);X_1\ X_2)=0.


	Suppose that (S;X_1\ X_2)=0. Then,
	I(S;X_1)=(S;X_1,X_2)=(f^*(S);X_1,X_2)≤ I(f^*(S);X_1) ≤ I(S;X_1). Thus, the data
	processing inequality holds with equality. This implies that X_1 - f^*(S) - S is a Markov chain. The identity
	(f^*(S);X_1\ X_2)=0 follows from the same chain of inequalities.



	
	If  has the Blackwell property, then  does not have the Blackwell property.


	As shown in the example in the appendix,
	
	there exist random variables S, X_1, X_2 and a function f that satisfy
	
		
  * S and X_1 are independent given f(S).
		
  * The channel f(S)→ X_1 is a garbling of the channel f(S)→ X_2.
		
  * The channel S→ X_1 is not a garbling of the channel S→ X_2.
	
	We claim that f solves the optimization problem (<ref>).  Indeed, for an arbitrary function f',
	
    (f'(S);X_1,X_2)≤ I(f'(S);X_1)≤ I(S;X_1) = I(f(S);X_1) = (f(S);X_1,X_2).

	Thus, f solves the maximization problem (<ref>).
	
	If  satisfies the Blackwell property, then (2) and (3) imply (f(S);X_1\ X_2) = 0 and (S;X_1\ X_2) > 0.  On the other hand,
	
    (S;X_1∖ X_2) 
    	= I(S;X_1) - (S;X_1,X_2) 
    	= I(S;X_1) - (f(S);X_1,X_2) 
    
    	= I(S;X_1) - I(f(S);X_1) + (f(S);X_1\ X_2)
    	
    	= 0.

	Thus,  does not satisfy the Blackwell property.



	There is no bivariate information decomposition in which  satisfies the Blackwell property and  satisfies
	left monotonicity.


	If  satisfies left monotonicity, then =.  Thus, = cannot satisfy the Blackwell
	property by Theorem <ref>.




§ LEFT MONOTONIC INFORMATION DECOMPOSITIONS



Is it possible to have an extractable information decomposition?  More precisely, is it possible to have an information decomposition in which all information measures are left monotonic?  The obvious strategy of starting with an arbitrary information decomposition and replacing each partial information measure by its extractable analogue does not work, since this would mean increasing all partial information measures (unless they are extractable already), but then their sum would also increase.  For example, in the bivariate case, when  is replaced by a larger function , then  needs to be replaced by a smaller function, due to the constraints (<ref>) and (<ref>).

As argued in <cit.>, it is intuitive that  be left monotonic. As argued above (and in <cit.>), it is also desirable that  be left monotonic. The intuition for synergy is much less clear. 
In the following, we restrict our focus to the bivariate case and study the implications of requiring both  and  to be left monotonic.
Proposition <ref> gives bounds on the corresponding  measure.

	
	Suppose that ,  and  define a bivariate information decomposition, and suppose that  and
	 are left monotonic.  Then,
	
    (f(X_1,X_2);X_1,X_2) ≤ I(X_1;X_2)

	for any function f.

Before proving the proposition, let us make some remarks. 
Inequality (<ref>) is related to the identity axiom.  Indeed, it is easy to
derive Inequality (<ref>) from the identity axiom and from the assumption that  is left monotonic. 
Although Inequality (<ref>) may not seem counterintuitive at first sight, none of the
information decompositions proposed so far satisfy this property (the function I_⋏
from <cit.> satisfies left monotonicity and has been proposed as a measure of shared information, but it does not lead to a nonnegative information decomposition).


	If  is left monotonic, then
	
    (f(X_1,X_2);X_1,X_2)    ≤(Copy(X_1,X_2);X_1,X_2) 
       = I(Copy(X_1,X_2);X_1) - (Copy(X_1,X_2);X_1\ X_2).

	If  is left monotonic, then
	
    (Copy(X_1,X_2);X_1\ X_2) ≥(X_1;X_1\ X_2) 
    	= I(X_1;X_1) - (X_1;X_1,X_2).

	Note that I(X_1;X_1) = H(X_1) = I(Copy(X_1,X_2);X_1) and 
    (X_1;X_1,X_2)=I(X_1;X_2)-(X_1;X_2\ X_1)=I(X_1;X_2).
 Putting these inequalities together, we obtain (f(X_1,X_2);X_1,X_2) ≤ I(X_1;X_2). 





§ EXAMPLES



In this section, we apply our construction to Williams and Beer's measure, I_min <cit.>, and to the bivariate measure of shared information, , proposed in <cit.>.  

First, we make some remarks on how to compute the extractable information measure (under the assumption that one knows how to compute the underlying information measure itself). The optimization problem (<ref>) is a discrete optimization problem. The search space is the set of functions from the support  of S to some finite set '.  For the information measures that we have in mind, we may restrict to surjective functions f, since the information measures only depend on events with positive probabilities.  Thus, we may restrict to sets ' with |'|≤||.
Moreover, the information measures are invariant under permutations of the alphabet .  Therefore, the only thing that matters about f is which elements from  are mapped to the same element in '.  Thus, any function f:→' corresponds to a partition of , where s,s'∈ belong to the same block if and only if f(s)=f(s'), and it suffices to look at all such partitions. The number of partitions of a finite set  is the Bell number B_||.

The Bell numbers increase super-exponentially, and for larger sets , the search space of the optimization problem (<ref>) becomes quite large.  For smaller problems, enumerating all partitions in order to find the maximum is still feasible.  For larger problems, one would need a better understanding about the optimization problem.
For reference, some Bell numbers include:


	
		n         3     4     6     10 

		
		B_n     5     15     203     115975
	.


As always, symmetries may help, and so in the Copy example discussed below, where ||=4, it suffices to study six functions instead of B_4 = 15.

We now compare the measure I_min, an extractable version of Williams and Beer's measure I_min (see Equation (<ref>) above),
to the measure , an extractable version of the measure  proposed in <cit.>.  
For the latter, we briefly recall the definitions. 
Let Δ be the set of all joint distributions of random variables (S,X_1,X_2) with given state spaces , _1, _2. Fix P=P_SX_1X_2∈Δ. Define Δ_P as the set of all distributions Q_SX_1X_2 that preserves the marginals of the pairs (S,X_1) and (S,X_2), that is,

    Δ_P{Q_SX_1X_2∈Δ: Q_SX_1=P_SX_1   , Q_SX_2=P_SX_2,∀ (S,X_1,X_2)∈Δ}.

Then, define the functions

    (S;X_1\ X_2)    min_Q∈Δ_P I_Q(S;X_1|X_2),
    (S;X_2\ X_1)    min_Q∈Δ_P I_Q(S;X_2|X_1),
    (S;X_1,X_2)    max_Q∈Δ_P CoI_Q(S;X_1,X_2),
    (S;X_1,X_2)     I(S;X_1X_2) - min_Q∈Δ_P I_Q(S;X_1X_2),

where the index Q in I_Q or CoI_Q indicates that the corresponding quantity is computed with respect to the joint distribution Q. 
The decomposition corresponding to  satisfies the Blackwell property and the identity axiom <cit.>. 
 is left monotonic, but  is not <cit.>. In particular, ≠.   can be characterized as the smallest measure of shared information that satisfies property (*).  Therefore,  is the smallest left monotonic measure of shared information that satisfies property (*).

Let =={0,1} and let X_1, X_2 be independent uniformly distributed random variables. Table <ref> collects values of shared information about f(X_1,X_2) for various functions f (in bits).


The function f_1:{00,01,10,11}→{0,1,2} is defined as






    f_1(X_1,X_2) :=
    
    X_1,     if X_2=1, 
    
    2,     if X_2=0.

The Sum function is defined as f(X_1,X_2) := X_1 + X_2.

Table <ref> contains (up to symmetry) all possible non-trivial functions f. 
The values for the extractable measures are derived from the values of the corresponding non-extractable measures.  
Note that the values for the extractable versions differ only for Copy from the original ones. In these examples, I_min=I_min, but as shown in <cit.>, I_min is not left monotonic in general.





§ CONCLUSIONS


We introduced a new measure of shared information that satisfies the left monotonicity property with respect to local operations on the target variable. Left monotonicity corresponds to the idea that local processing will remove information in the target variable and thus should lead to lower values of measures which quantify information about the target variable. Our measure fits the bivariate information decomposition framework; that is, we also obtain corresponding measures of unique and synergistic information.
However, we also have shown that left monotonicity for the shared information contradicts the Blackwell property of the unique information, which limits the value of a left monotonic measure of shared information for information decomposition. 

We also presented an alternative interpretation of the construction used in this paper. Starting from an arbitrary measure of shared information  (which need not be left monotonic), we interpret the left monotonic measure  as the amount of shared information that can be extracted from S by local processing.

Our initial motivation for the construction of  was the question to which extent shared information originates from the redundancy between the predictors X_1 and X_2 or is created by the mechanism that generated S.
These two different flavors of redundancy were called source redundancy and mechanistic redundancy, respectively, in <cit.>.
While  cannot be used to completely disentangle source and mechanistic redundancy, it can be seen as a measure of the maximum amount of redundancy that can be created from S using a (deterministic) mechanism.
In this sense, we believe that it is an important step forward towards a better understanding of this problem and related questions.




















§ APPENDIX: COUNTEREXAMPLE IN THEOREM <REF>
  



Consider the joint distribution


	
		f(s)     s   x_1   x_2    P_f(S)SX_1X_2 

		0     0     0     0     1/4 

		
		
		0     1     0     1     1/4 

		0     0     1     0     1/8 

		0     1     1     0     1/8 

		
		1     2     1     1     1/4 

	


and the function f:{0,1,2}→{0,1} with f(0)=f(1)=0 and f(2)=1. Then, X_1 and X_2 are independent uniform binary random variables, and f(S) = And(X_1,X_2). In addition, S-f(S)-X_1 is a Markov chain. By symmetry, the joint distributions of the pairs (f(S), X_1) and (f(S), X_2) are identical, and so the two channels f(S)→ X_1 and f(S)→ X_2 are identical, and, hence, trivially, one is a garbling of the other. However, one can check that the channel S→ X_1 is not a garbling of the channel S→ X_2.



















































This example is discussed in more detail in <cit.>.










IEEEtran


