A conditional independence framework for coherent modularized inference
Manuele Leonelli Martine J. Barons Jim Q. Smith




#1 1




1
 
  
  A conditional independence framework for coherent modularized inference
    Manuele Leonelli 

   School of Mathematics and Statistics, University of Glasgow




    Martine J. Barons and Jim Q. Smith 

   Department of Statistics, University of Warwick
    December 30, 2023
=====================================================================================================================================================================================

 

0
  

    A changepoint approach for the identification of financial extreme regimes


   


Inference in current domains of application are often complex and require us to integrate the expertise of a variety of disparate panels of experts and models coherently. In this paper we develop a formal statistical methodology to guide the networking together of a diverse collection of probabilistic models. In particular, we derive sufficient conditions that ensure inference remains coherent across the composite before and after accommodating relevant evidence.



Keywords: 
Coherence; Conditional independence; Likelihood separation; Modular inference.


1.45 


§ INTRODUCTION


For large scale applications is now often necessary to integrate many probabilistic components - themselves often massive - into a single coherent system. One of the earliest examples comes from nuclear emergency management. There engineering, athmospheric, economic and social components needed to be integrated to give uncertainty measures of the possible unfoldings of a crisis <cit.>. Then expedience dictated that point estimates of outputs from one component was donated to an adjacent component whilst the uncertainty associated with such estimates was simply discarded. This naive approach of course dramatically affects the coherence of the composite system both for decision making <cit.> and inference <cit.>. Despite of the dangers associated to modular approaches, such practice is still often used, for instance in climate modelling <cit.>, air pollution  <cit.>, econometrics <cit.> and epidemiology <cit.>.

Exceptions of course exist. For example some early work for the full integration of different components was developed in artificial intelligence <cit.> although this work exclusively focused on  Bayesian networks. Only recently has the statistical community recognized the need for a more general study of how and when such an integration can be coherently performed <cit.>. In the context of graphical models, Markov combinations <cit.> and Markov melding <cit.> are coherent frameworks designed to compose models which share some common features. <cit.> characterizes the cases where a modular approach may outperform the inference resulting from a full joint statistical specification.

Here we consider the situation where inference can be exclusively carried out in a modular fashion, where different panels of experts are responsible for their own component of the system which they update autonomously. In the above-mentioned applications, the fields of expertise that need to be brought together are so heterogeneous that no-one could realistically own all the statements about the fully joint distribution unless there existed shared structural assumptions. Even if it were feasible to build a single joint model, typically the components need to be continually revised to accommodate new undersanding, science and data. Any overaching probabilistic model would thus quickly become obsolete since the judgements it embodied would no longer reflect current understanding.

Here we determine when and how expert judgement and data in such a modular framework can be formally and coherently combined, taking into account of the relative strength of different evidence. We demonstrate that under a set of structural assumption often met in practice, inference can be guaranteed to be coherent and also show that the beliefs it embodies can be structured in a way that incorporates  all usable, informed expert judgment and data drawn together from the different components. Furthermore, in a sense we define later, if an appropriate protocol guiding the nature and quality of the data input is in place, then inference can be distributed, meaning that that the system remains coherent when each panel  autonomously  updates its beliefs about its domain of expertise. Distributivity gives the added benefit that any output probability associated with the composite can be typically calculated very quickly <cit.>.




§ THE CONDITIONAL INDEPENDENCE FRAMEWORK



We start setting up a formal framework for the combination of panels' judgements and components. A large vector of random variables, Y, measures various features of an unfolding future. Suppose that different components of this vector are evaluated and overseen by m different panels of domain experts, G_1,…, G_m. Panel G_i is responsible for the output vector Y_i, where Y_i takes values in 𝕐_i, i∈[m]={1,…,m}, and Y=( Y_i)_i∈[m].  The implicit (albeit virtual) owner of the beliefs of the panels is henceforth referred to as the supraBayesian, meaning that the group reasons as a single person would and it is her coherence that we are concerned with. 

A modular system should endeavour to accommodate probabilistic information provided by the most well-informed experts only. Therefore each panel only donates beliefs associated with their own domain of expertise and not their beliefs about the whole vector Y. Each panel G_i is asked to give the supraBayesian their beliefs about the probability distribution of Y_i over which G_i has oversight, conditional on certain measurable functions { A_i( Y):A_i∈𝔸_i}, where 𝔸_i could be null. For instance 𝔸_i could be the set of different possible combinations of the covariates' levels on which Y_i might depend. 

Henceforth we assume that all panels make their inferences in a parametric setting where Y is parametrised by θ=( θ_i)_i∈[m]∈Θ, and θ_i parametrises  G_i's sample distributions. When the parameter space can be written as a product space, Θ = ×_i∈[m]Θ _i,  where Θ_i is G_i's parameter space, panels are variationally independent <cit.>. We  assume this property holds, as often in practice. This is in fact a necessary condition for inference to be distributed.

In this parametric setting panels  have two quantities available to them. The first are sample distibutions over future measurements for which they have responsibility 
Π _i^y|θ≜{f_i(Y_i | A_i, θ_i):θ_i∈Θ _i, A_i ∈𝔸_i}. For example, if Y were discrete and finite, then each panel might be asked to provide certain multi-way conditional probability tables over their subvector Y_i, conditional on each A_i∈𝔸_i. In this case θ_i would be the concatenated probabilities within all these tables. The second are panel densities
Π _i^θ≜{π _i( θ_i | A_i):A_i ∈𝔸_i}, such as a joint probability distribution over all the probabilities specified in the multi-way conditional probability tables above. Of course panels may want to use various data available to them to infer the distribution of θ_i, i∈[n]. However each panel typically performs this inference autonomously. 

If the supraBayesian's beliefs are functions of the autonomously calculated panels' beliefs we say that inference is distributed.

If G_i's inferences are going to be inherited by the supraBayesian, it is by no means automatic that such autonomous updating is justified.  We henceforth assume Property <ref> holds.



All agree on the variables Y defining the process together with qualitative statements about the dependence between Y and θ. Call these assumptions the structural consensus.


This consensus can often be expressed through an appropriately chosen graphical or conditional independence structure both across  the distribution of Y | θ, and also that of θ.  These types of assumptions are often complex but can be represented by a variety of well-known frameworks <cit.>. Technically we can think of the structural consensus as the qualitative beliefs that are shared as common knowledge by all the panel members. These are usually expressed in common  language and so are more likely to form part of the structural consensus <cit.>.  For the coherence of modular systems, inference also needs to be sound.


If by adopting the structural consensus the supraBayesian underlying beliefs about a domain overseen by a panel G_i are {Π _i^y|θ,Π _i^θ}, i∈[m], we say that inference is sound.




§ ILLUSTRATIONS OF SOME OF THE INFERENTIAL CHALLENGES


We next give a flavour of the difficulties associated with the integration of the panels beliefs and the dangers of modular inference when not justified. Consider the simplest scenario where  the number of panels m=2 and the structural consensus specifies that Y= (Y_1,Y_2) are binary. Here the random variable Y_1 is an indicator of whether or not a foodstuff has become poisonous and  Y_2 is an indicator of whether or not contamination is detected before the food is distributed to the public. Sample distributions Π_i^y|θ given by expert panel G_i are saturated so that θ_i≜ P(Y_i=1). Suppose the structural consensus further specifies that Y_2 Y_1 | (θ_1,θ _2).  This implies that, given the probabilities θ_1 and θ_2, observing poisonous foodstuff does not affect our beliefs about the detection regime. Suppose the main inferential object of interest is 
 𝔼(Y_1Y_2), an indicator of whether the public is exposed to the contamination. To derive such score the supraBayesian needs to able to calculate 𝔼( θ_1θ _2). This can be derived from the panel's densities if and only if the independence θ _1θ _2 was in the structural consensus. With this additional condition, assume a random vector ( X_1,X_2) is sampled from the same population as ( Y_1,Y_2) so that each panel updates  its parameter densities from π _i(θ _i ) to π_i(θ _i | x_i), i∈[2], using its own separate randomly sampled populations, x_i, concerning θ_i alone. The supraBayesian inference is then sound and distributed. By adopting all these beliefs as her own, she acts as if she had sight of all the available information and had processed this information herself.


Note that in the example above  the  assumption θ_1θ_2 is critical for distributivity. Suppose on the contrary that  π (θ _2 | θ _1)  needs to be a function of θ _1. Then
π _2(θ _2 | x_1,x_2)=∫_0^1π _2(θ _2 | θ_1,x_2)π_1(θ _1 | x_1)θ _2,
 where the prior dependence of θ_2 on θ_1 induces a dependence of θ_2 on x_1. So  π _2(θ _2 | x_1,x_2)≠π _2(θ _2 | x_2) in general. By devolving inference to the two panels who learn autonomously,  the supraBayesian cannot act as a single Bayesian would by using π _2(θ _2 | x_2). It follows that the system is no longer sound, although when supporting evidence remains unseen the supraBayesian  appears to act coherently.  She is, implicitly,  assuming that θ_1 θ_2, which is contrary to the reasoning G_2 would want to provide.


Perhaps more importantly, even if global independence is justified a priori, the assumption that data collected by the two panels and individually used to adjust their beliefs does not inform both parameters is also a critical one.  Suppose that G_1 and G_2 both see their respective margin concerning the experiment in Table <ref> and each uses this experiment to update its  marginal distribution on θ _i, i∈[2].
 
If both began with a prior symmetric about 0.5 and the structural consensus included θ _1θ _2 then the supraBayesian would assign 𝔼(Y_1Y_2 | x_1,x_2)=𝔼(Y_1 | x_1)𝔼(Y_2 | x_2)=0.25.
This inference contrasts with inferences the supraBayesian would make on seeing the whole table and assuming θ _1θ _2 a priori. With a fairly uninformative prior on the two margins, her judgement would be approximately 0.05, five times smaller than above. If conversely it were only possible to see the table of randomly sampled counts associated with Y_1Y_2 and the supraBayesian used this information directly, for example by introducing a uniform prior on P(Y_1Y_2=1), then her  posterior mean would be approximately 0.05. However, observations have made the  independence θ_1θ_2 no longer formally valid and the distributivity of the system is destroyed if this data is accommodated. 

Thus considerable care needs to be exercised before a modular system can be expected to work reliably, even in the simplest of networks.  In the next section we  prove some conditions which ensure the supraBayesian inference is distributed and sound.  



§ COHERENCE AND MODULARIZATION



 §.§ Information and admissibility protocols


Suppose panels are progressively informed by new collections of evidence.  In practice, within the totality of information conceivably available at time t, usually only a subset, the admissible evidence, is of sufficient quality and has suitable form to be used. The sort of information excluded or delayed admittance might include evidence whose relevance is ambiguous or of a type which might introduce insurmountable computational challenges. An admissibility protocol is therefore needed to avoid the difficulties illustrated in Section <ref>. 

Let I_0^t denote all the admissible evidence which is common knowledge to all panel members at time t. Let I_ij^t denote the subset of the admissible evidence panel G_i would use at time t if acting autonomously to assess their beliefs about θ_j,  i,j∈[m].  We define the admissible evidence as I_+^t≜{ I_ij^t:i,j∈[m]}∪ I_0^t and I_∗^t≜{ I_ii^t:i∈[m]} to be the subset of the admissible evidence each panel G_i would use to update θ_i, i∈[m].

Although not explicitly stated, protocols for the selection of good quality evidence are often used  in several domains. A notable example is the Cochrane systematic review <cit.> developed to pare away information which might be
ambiguous and potentially distort evidence concerning medical treatments, through a trusted set of principles relevant to the domain. This may seem restrictive, but in practical applications the need to be selective about experiments that can provide evidence of an acceptable quality is  widely acknowledged. 

For a modular system, the needs for such an admissibility protocol are even more important
because of its collective structure. So we assume that panels, both individually and corporately, agree on an appropriate protocol for selecting suitable evidence, mirroring Cochrane reviews in ways relevant to their domain. However, for our particular purposes one additional requirement is needed in this setting: the chosen admissibility protocol must also ensure distributivity. For we have already argued that, if this is not the case, then the supraBayesian inference is either dependent on arbitrary assumptions and difficult to calculate or, if distributivity is forced, incoherent. 



 §.§ Conditional independence conditions for coherence


Assume that all panellists, as represented collectively by the supraBayesian, agree that their inferences should obey the (qualitative) semi-graphoid axioms. These axioms are widely accepted as appropriate for reasoning about evidence when irrelevance statements are read as conditional independences: Bayesian systems always respect these properties <cit.>. The four properties below ensure the soundness of a supraBayesian inference within an admissibility protocol. Although these are in no sense automatic, they are, nevertheless,  satisfied by a very diverse collection of models and information sets and can be checked for their plausibility in any given context <cit.>. Let θ_i^-≜( θ _j)_j∈[m]∖{i}.  



An admissibility protocol is said delegable at time t if 

    I_+^tθ | I_0^t,I_∗^t,

separately informed at time t if 

    I_ii^tθ_i^- | I_0^t,θ_i,

cutting at time t if 

    I_∗^tθ_i | I_0^t,I_ii^t,
    θ_i^-,

commonly separated at time t if 

    _i∈[m]θ_i | I_0^t.






An admissibility protocol is said:


  * delegable at time t if I_+^tθ | I_0^t,I_∗^t;

  * separately informed at time t if 
I_ii^tθ_i^- | I_0^t,θ_i;

  * cutting at time t if I_∗^tθ_i | I_0^t,I_ii^t,θ_i^-; 

  * commonly separated at time t if _i∈[m]θ_i | I_0^t.  




 An admissibility protocol is delegable when the admissible evidence I_+^t is the union of the evidence I_0^t shared by all panels plus the evidence I_∗^t each panel has about its own domain of expertise. This condition is ensured if panels are working collaboratively rather than competitively. Alternatively the protocol could itself simply demand that I_+^t={I_0^t,I_∗^t}.
 
  For the separately informed condition, pieces of evidence G_i might collect individually cannot be informative about θ_i^- once the domain experts' evidence has been fed in. This is satisfied when a new piece of information could only be added to I_ii^t  if the evidence it provided about θ_i would not depend on θ_i^-. Although commonly valid <cit.>, one context where  it is violated is when sampling is not ancestral <cit.>: sometimes due to a hidden confounder across the different models of the system.

For cutting, once { I^t_0,I_ii^t} is known, no-one believes that a panel G_j has used any information that G_i might also want to use to adjust its beliefs about θ_i. This captures what we might mean by `panel of experts'. One situation when the cutting condition is violated is if  a panel G_i marginalises out a parameter in θ_j to accommodate a piece of evidence, making θ_i and θ_j dependent on each other. 

When parameters are commonly separated the information that everyone shares separates the parameters. We note that this  is nearly always  assumed in the context of practical applications of graphical models <cit.>. 

The following result, analogous to <cit.> about the use of linear Bayes methods and a single agent, can now be proved. 


Under structural consensus, the supraBayesian inference is sound and distributed if the admissibility protocol is delegable, separately informed, cutting and commonly separated. 




The result follows if, at any time t, the supraBayesian holds panel independent beliefs

    θ_iθ_i^- | I_+^t,

and if, when assessing θ_i, she only uses the information G_i would use if acting autonomously

    θ_i I_+^t | I_0^t,I_ii^t.

These conditions are then sufficient for soundness and distributivity. Because even if all panels could share each other's information then they would come to the same assessment. 

To prove condition (<ref>) note that from common separability it follows that θ_i θ_i^- |  I_0^t, which combined with the separately informed condition in equation (<ref>) through perfect decomposition and using the symmetric property of semi-graphoids axioms gives θ_i^- I_ii^t,θ_i |  I_0^t.
Using perfect decomposition and symmetry again, it follows that θ_iθ_i^- |  I_ii^t,I_0^t. When combined with the cutting condition in (<ref>) via perfect decomposition this gives 

    θ_i I_*^t,θ_i^- |  I_0^t,I_ii^t.

 Then again by perfect decomposition and since I_ii^t is a function of I_*^t we have that

    θ_iθ_i^- |  I_0^t, I_*^t.

Using perfect decomposition, the delegatable condition in equation (<ref>) can be written as

    I_+^tθ_i |  I_0^t,I_*^t,θ_i^-.

Combining via perfect decomposition equations (<ref>-<ref>) and using symmetry we have that θ_i I_+^t,θ_i^- |  I_0^t,I_*^t, so by perfect decomposition it follows that θ_iθ_i^- |  I_0^t,I_*^t,I_+^t, 
which, since I_*^t and I_0^t are functions of I_+^t, can be written as equation (<ref>). 

To show that equation (<ref>) holds, note that another implication of delegatability in equation (<ref>) by perfect decomposition is that, again using the fact that I_ii^t is a function of I_*^t,

    θ_i I_+^t |  I_0^t,I_ii^t,I_*^t.

Noting that by perfect decomposition equation (<ref>) implies that

    θ_i I_*^t  |  I_0^t,I_ii^t,

from equations (<ref>) and (<ref>) by perfect decomposition it follows that θ_i   I_+^t,I_*^t |  I_0^t,I_ii^t,. This, since I_*^t is a function of I_+^t, is equivalent to equation (<ref>).




 §.§ Likelihood separation for modularized inference


We now focus our attention onto  probabilistic systems. Suppose that the supraBayesian believes  _i∈[m]θ_i | I_+^0 and that the only evidence available to panels is in the form of datasets x^t={x_τ:τ≤ t} which then populate  I_+^t. The features that ensure soundness and distributivity can be expressed in terms of the separability of a likelihood. Let l(θ | x^t) denote a likelihood over the parameter θ of the distribution of Y given x^t. 


An admissibility protocol is said panel separable if

    l(θ | x^t)=∏_i∈[m]l_i(θ_i | t_i(x^t)),

where l_i(θ_i | t_i(x^t)) is a function of θ only through θ_i and t_i(x^t) is a statistic of x^t  formally accommodated by G_i into I_ii^t to form its own posterior assessment of θ_i, i∈[m].


We now have the following theorem that gives good practical guidance about when and how soundness and delegability can be preserved over time, as new datasets inform the composite.


Under structural consensus, suppose the supraBayesian inference is sound and distributed at time t=0. Provided the prior over θ is absolutely continuous with respect to Lebesgue measure, the supraBayesian inference is sound and distributed at time t>0 if and only if the admissibility protocol is panel separable. 



Under the initial hypotheses, by Theorem <ref>, the prior joint density can be written in a product form  π (θ)=∏_i∈[m]π _i(θ_i). It follows that under this admissibility protocol
π (θ | x^t)=∏_i∈[m]π_i(θ_i,t_i(x^t)), 
where π _i(θ_i,t_i(x^t))∝ l_i(θ_i | t_i(x^t))π _i(θ_i). By hypothesis π _i(θ_i,t_i(x^t)) is adopted by the supraBayesian, thus guaranteeing soundness. In particular we have that 

    _i∈[m]θ_i | I_0^t,I_+^t  ⟺  _i∈[m]θ_i | I_0^t,I_∗^t,

where I_+^t ={I_0^t,I_∗^t} ={ I_0^0,I_∗^0,x^t} and, for i∈[m], { I_0^t,I_ii^t} ={ I_0^t,I_ii^0,t_i(x^t)}. Since by definition t_i(x^t) is known to G_i, i∈[m],  the system is also distributed. 

Finally note that if l(θ | x^t)≠∏_i∈[m]l_i(θ_i | t_i(x^t)) on a set A of non zero prior measure, then the posterior density π_A(θ | x^t) on A has the property that, for all θ∈ A,
π _A(θ | x^t)≠∏_i∈[m]π _A,i(θ_i,t_i(x^t)), 
where π_A,i denotes the density delivered by panel G_i for the parameters it oversees in A. This means that panel parameters are a posteriori dependent. So in particular the density determined by the margins is not sound.


By designing  a single experiment to be orthogonal over parameters θ_i  and θ_j, overseen by G_i and G_j respectively, inference is sound and distributed under the conditions of the theorem. From this single experiment  data can still be included in the admissible evidence. Note, however, that the converse demonstrates that some protocol is certainly needed to preserve  distributivity. 



§ CONCLUSION

The complexity of modern applications often require inference to be  modular. Here we have identified a set of conditions under which autonomous component distributions can be aggregated coherently to provide a unique probabilistic assessment. Many modelling tools commonly used, often in the form of a graphical model, can be set up to respect such conditions <cit.>. The novel framework we introduced here is currently being used in practice to address the critical application of food security, where a suite of diverse components need to be networked together in a coherent fashion to support policy makers <cit.>. Interestingly when such systems are only used to inform a formal Bayesian decision analysis, it is possible to show that the conditions that lead to soundness and distributivity can be made  milder than the ones required in the fully inferential settings discussed in Theorems <ref> and <ref> <cit.>.

Of course, in some situations the likelihood separation needed to ensure the enduring formal distributivity breaks down. Then we need to fall back on approximate inferential methods that preserve distributivity. However preliminary results suggest that sensible approximate methods, whose form is similar to variational Bayes methods, can still work effectively provided that statistical diagnostics are employed to check that the structural consensus remains plausible in the light of the information available. 



§ ACKNOWLEDGEMENT

Research supported by EPSRC, JQS & MJB were funded by EPSRC grant EP/K039628/1; JQS also supported by the Alan Turing Institute under EPSRC grant EP/N510129/.

Chicago


