 (DIM(L)) [@DIM] as a variant of MMI predictive coding.

A closely related framework is the information bottleneck [@bottleneck]. Here one again assumes a population distribution on pairs $(x,y)$. The objective is to learn a stochastic coding function $C_x$ so as to maximize $I(C_x(x),y)$ while minimizing $I(C_x(x),x)$. Here one does not ask for a coding function on $y$ and one does not limit $H(C_x(x))$.

Another related framework is INFOMAX [@linsker1988self; @bell1995information; @DIM]. Here we consider a population distribution on a single random variable $x$. The objective is to learn a stochastic coding function $C_x$ so as to maximize the mutual information $I(x,C_x(x))$ subject to some constraint or additional objective.

As mentined above, in cases where $I(C_x(x),C_y(y))$ is large it seems best to train a model of the marginal distribution of $P(C_y)$ and a model of the conditional distribution $P(C_y|C_x)$ where both models are trained with cross-entropy loss. Section \[sec:cross\] gives various high confidence upper bounds on cross entropy loss for learned models. The main point is that, unlike lower bounds on entropy, high-confidence upper bounds on cross entropy loss can be guaranteed to be close to the true cross entropy.

Our theoretical analyses will assume discrete distributions. However, there is no loss of generality in this assumption. Rigorous treatments of probability (measure theory) treat integrals (either Reimann or Lebesque) as limits of increasingly fine binnings. A continuous density can always be viewed as a limit of discrete distributions. Although our proofs are given for discrete case, all our formal limitations on the measurement of mutual information apply to continuous case as well. See [@diffent] for a discussion of continuous information theory. Additional comments on this point are given in section \[sec:limitations2\].

The Donsker-Varadhan Lower Bound
================================

Mutual information can be written as a KL divergence. $$I(X,Y) = KL(P_{X,Y},P_XP_Y)$$ Here $P_{X,Y}$ is a joint distribution on the
(DIM(L) ) [ @DIM ] as a variant of MMI predictive coding. 

 A closely related model is the data constriction [ @bottleneck ]. Here one again assumes a population distribution on pairs $ (x, y)$. The objective is to teach a stochastic coding function $ C_x$ so as to maximize $ I(C_x(x),y)$ while minimize $ I(C_x(x),x)$. Here one does not ask for a coding affair on $ y$ and one does not limit $ H(C_x(x))$. 

 Another related model is INFOMAX [ @linsker1988self; @bell1995information; @DIM ]. Here we consider a population distribution on a single random variable $ x$. The objective is to determine a stochastic coding function $ C_x$ so as to maximize the mutual information $ I(x, C_x(x))$ subject to some restraint or additional objective. 

 As mentined above, in cases where $ I(C_x(x),C_y(y))$ is big it seems best to coach a model of the fringy distribution of $ P(C_y)$ and a model of the conditional distribution $ P(C_y|C_x)$ where both models are trained with cross - entropy loss. incision   \[sec: cross\ ] gives various high confidence upper bounds on cross entropy loss for learned models. The main point is that, unlike lower bounds on entropy, high - confidence upper bounds on crisscross entropy loss can be guarantee to be cheeseparing to the true cross entropy. 

 Our theoretical analyses will assume discrete distributions. However, there constitute no loss of generality in this assumption. Rigorous treatments of probability (measuring stick theory) treat integrals (either Reimann or Lebesque) as limits of increasingly all right binnings. A continuous density can always be viewed as a limit of discrete distributions. Although our proofs are given for discrete font, all our formal limitations on the measurement of mutual information apply to continuous shell as well. See [ @diffent ] for a discussion of continuous information hypothesis. Additional comment on this point are given in department   \[sec: limitations2\ ]. 

 The Donsker - Varadhan Lower Bound 
 = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = 

 Mutual information can be spell as a KL divergence. $ $ I(X, Y) = KL(P_{X, Y},P_XP_Y)$$ Here $ P_{X, Y}$ is a joint distribution on the
 (DIL(L)) [@DIM] as a variant of MOI predictive coding.

A rlosely related framework is the informatioi bortlentbk [@bottleneck]. Here ond again adsumes a popnlation distribuvjon on icirs $(s,n)$. The ibjective is tp learn a vtochastic codhne yunction $C_x$ so as to maximize $I(C_x(x),y)$ rhile monlmizing $I(C_x(x),x)$. Rere jne soes not ask for a coding function on $y$ aid one does not limit $H(C_x(x))$.

Another related fralework is INFOMAX [@pinsker1988self; @belj1995unformation; @AIM]. Here wt eonsider a kopulation distribution on a sinele rcndom variavlw $x$. Jhe objectivx is tj learn a stochastic coding function $C_x$ sp av ti maximize the mutual information $I(x,C_x(x))$ subject do some constraint ir addijionan obgwctkve.

Zs msntinef auove, in casss where $I(C_z(x),C_y(y))$ is large it stemf best to traih a moqej of the marginal distribution of $P(C_y)$ atd z model of the conditiobal distribution $P(C_y|C_d)$ where bjth models are trained with cross-entropy loss. Secdion \[sxc:zrows\] euvfs various high confidence upper bounds on crjas ektropy loss for jearned modrld. Yre main point is thcf, hnlike lower boundd on enjropy, yigh-confiqencr upper bounds on cross enteopy loss cau bw guaranteed to be close to tke trug crosx entropy.

Our theoreticau anzlyses will assume djrcrete distributkonx. Vowever, ufere is no loss os generalmty iu this arsumktion. Rygorous trfatmekds of probability (leasute themry) treat lntegrals (either Reimann or Lebxxque) as limijs mf pncreasinyly fike binnings. A cjntinuous densnty can clways be viewed as a lmmit of discwete distribudlons. Althougi our projfs qre tiven fuf discrete casr, all our formal linitations on the mcasurgmsnt of mutual iuyotnation apply tp cuntynloux sdse as well. Vee [@aifwrnt] fur a discuswkon pf continuous informdtioh theory. Additionak gomments in this [oint are givrn in section \[sec:lilitatmons2\].

Thx Donsler-Daradhan Lower Bound
================================

Mutual infkrmation fan be written ws a KL divergenee. $$I(X,Y) = KL(P_{X,Y},P_XP_Y)$$ Here $P_{X,Y}$ is a joint diwtribution on the
(DIM(L)) [@DIM] as a variant of MMI A related framework the information bottleneck a distribution on pairs The objective is learn a stochastic coding function $C_x$ as to maximize $I(C_x(x),y)$ while minimizing $I(C_x(x),x)$. Here one does not ask for coding function on $y$ and one does not limit $H(C_x(x))$. Another related framework INFOMAX @bell1995information; Here consider a population distribution on a single random variable $x$. The objective is to learn a coding function $C_x$ so as to maximize the information $I(x,C_x(x))$ subject to constraint or additional objective. As above, cases where is it best to train model of the marginal distribution of $P(C_y)$ and a model of the conditional distribution $P(C_y|C_x)$ where both are trained loss. Section gives high upper bounds on loss for learned models. The main unlike lower bounds on entropy, high-confidence upper bounds cross entropy can be guaranteed to be close the true cross entropy. Our theoretical analyses will discrete distributions. However, there is no loss of generality in this assumption. Rigorous treatments of theory) treat integrals (either or Lebesque) as of fine A density can be viewed as a limit of discrete distributions. Although our proofs given for discrete case, all our formal limitations on the mutual apply to continuous as well. See [@diffent] a of continuous information theory. on point section The Lower Bound ================================ Mutual can be written as a divergence. $$I(X,Y) = KL(P_{X,Y},P_XP_Y)$$ distribution on the
 (DIM(L)) [@DIM] as a variant of MMI preDictive codIng.

A cLosEly ReLateD fraMework is the infORmatIon bottleneck [@bottlenecK]. Here OnE AgaiN AsSumes A populaTIoN DIstRiBuTioN oN PaIrs $(x,y)$. the ObjectiVe is to learN a sToChastic codinG FuNction $C_x$ so As tO maximize $I(C_x(X),y)$ wHile miNiMizINg $I(C_x(X),x)$. HEre onE does nOT ask foR a coding fUnCTion on $Y$ And one dOES nOt liMit $H(C_x(x))$.

Another relATeD Framework is INFoMAX [@liNsKEr1988SELf; @bEll1995InformatioN; @DiM]. HerE We consiDEr A POPulATion distributIon on a singlE RanDom varIaBle $X$. the objEctivE iS To lEarn a stochaStic Coding funCtion $C_X$ So as to mAXimize tHe mutuAl iNfoRmatIOn $i(x,c_x(x))$ SuBJecT To SomE ConStraint oR aDdItionAl obJECTIve.

AS meNtinEd aboVe, in cases wherE $I(C_X(x),C_y(Y))$ Is lArge iT seemS besT tO traiN a modeL of thE mArginal distribuTion Of $P(C_y)$ and a ModEl Of tHe CondiTIonal dIstRibUtion $P(C_Y|C_x)$ wherE BotH mODELs Are trained with crosS-eNTRoPy loss. SeCtion \[sEC:cRoSS\] gives vaRiOus High CONfideNce uPPeR bounds oN cross ENtRoPy loss fOr LearneD mOdeLs. THe maiN PoinT is thaT, unlike lOwer bOUnds on entropy, hIGh-confidence uPPeR BOuNDs on CroSs entropy loSs caN Be guAranTEeD to BE closE to thE tRUe CRoss entropy.

Our theorEtIcal anAlyseS will assume diScrete distRIBUtions. HoWeveR, ThERe is no loss of geNeralIty in this aSSumption. rigorOus treatMents of prOBAbility (mEasUre TheOry) TREaT integrals (eitHER ReiMaNn or LebEsqUe) as limIts Of iNcrEasInGly fine biNnings. A cOnTiNuOuS deNsity CAn always Be VieWeD as A limiT Of discRete dIstrIbUtIOns. althougH OuR PRoofS aRe GiveN foR dIscreTe caSE, alL our forMal limitaTioNS on tHe MeAsuremeNt of mutual infOrMation applY tO coNtinuoUS Case as weLl. See [@diffent] for a discussIOn of conTinUous iNforMation theOry. additiOnaL CommenTs on thIs poiNt Are GIVen in SECtIon \[SeC:limitatioNS2\].

the donskEr-varaDhan LowEr Bound
================================

Mutual inforMAtiOn can be writteN as A KL dIVErGenCE. $$I(x,y) = KL(p_{X,y},p_XP_y)$$ hEre $P_{X,Y}$ is a joint dIstributioN oN ThE
 (DIM(L)) [@DIM] as a vari ant of MMI  pred ict ive  c odin g.
A closely rela t ed f ramework is the inform ation  b o ttle n ec k [@b ottlene c k] . Her eon e a ga i nassum esa popul ation dist rib ut ion on pairs $( x,y)$. The  ob jective is t o l earn a  s toc h astic  co dingfuncti o n $C_x $ so as t om aximiz e  $I(C_x ( x ), y)$while minimizing$ I( C _x(x),x)$. Her e onedo e sn o t a skfor a codi ng  func t ion on$ y$ a n d o n e does not li mit $H(C_x( x ))$ .

Ano th err elated  fram ew o rkis INFOMAX[@li nsker1988 self;@ bell199 5 informa tion;@DI M].  Her e  w econ si d era  p opu l ati on distr ib ut ion o n as i n g le r and om v ariab le $x$. The o bje ctiv e  is  to l earna st oc hasti c codi ng fu nc tion $C_x$ so a s to  maximize  th emut ua l inf o rmatio n $ I(x ,C_x(x) )$ subj e ctto s o me  constraint or add it i o na l object ive.
A sme n tined ab ov e,in c a s es wh ere$ I( C_x(x),C _y(y)) $  i slarge i tseemsbe sttotrain a mo del of  the mar ginal distribution o f  $P(C_y)$ and am o de l  ofthe  conditiona l di s trib utio n  $ P(C _ y|C_x )$ wh er e  b o th models are train ed  withcross -entropy loss . Section\ [ s ec:cross \] g i ve s  various highconfi dence uppe r  boundson cr oss entr opy lossf o r learne d m ode ls.  Th e ma in point is t h a t, u nl ike low erboundsonent rop y,hi gh-confid ence upp er  b ou nd s o n cro s s entrop ylos scan  be g u arante ed to  becl os e  to  the tr u ec r ossen tr opy. 

O ur  theo reti c alanalyse s will as sum e  dis cr et e distr ibutions. How ev er, thereis  no  losso f  general ity in this assumption. Rigorou s t reatm ents  of proba bil ity (m eas u re the ory) t reatin teg r a ls (e i t he r R ei mann or Le b e squ e) as  l imit s of in creasingly fine bi n nin gs. A continu ous  den s i ty  ca n  a l way sb e v i e wed as a limitof discret ed is tributions .  Al th ough ou r proof s are given f or discre te case,al l ou r for mal limita tions on  the meas u remen t  o f mut ual  infor ma tio n app ly toc ont inuou s case  a s well . See  [ @diffent ] for a discussion of c ontinu ous i nfo rmation t heo r y.Additiona l co mments onthi s p ointare given  ins ec tio n  \[se c:li m itations2 \ ]. 

T h e  D onsker-Vara d h a n L owerBou n d
==== ==== ================= = ======

Mutual  inf o r mat ion canbe  written as aKLdi v e rgence.$$ I(X,Y) = KL (P_{X,Y} ,P _ XP_Y) $$ Her e $P_{ X,Y}$ i s aj oint d istr ibu tion on t he
 (DIM(L))_[@DIM] as_a variant of MMI_predictive coding.

A_closely_related framework_is_the information bottleneck_[@bottleneck]. Here one_again assumes a population_distribution on pairs_$(x,y)$._The objective is to learn a stochastic coding function $C_x$ so as to maximize_$I(C_x(x),y)$_while minimizing_$I(C_x(x),x)$._Here_one does not ask for_a coding function on $y$_and one_does not limit $H(C_x(x))$.

Another related framework is INFOMAX_[@linsker1988self;_@bell1995information; @DIM]. Here_we consider a population distribution on a single random_variable $x$. The objective is to_learn a stochastic_coding_function_$C_x$ so as to_maximize the mutual information $I(x,C_x(x))$ subject_to some constraint or additional objective.

As_mentined above, in cases where $I(C_x(x),C_y(y))$ is_large it seems best to train_a model of the marginal_distribution of_$P(C_y)$ and a model of_the conditional distribution_$P(C_y|C_x)$ where_both models are_trained with cross-entropy loss. Section \[sec:cross\] gives_various high confidence_upper bounds on cross entropy loss_for_learned models. The_main_point_is that,_unlike lower bounds_on_entropy, high-confidence_upper_bounds on cross entropy loss can_be_guaranteed to be close to the true_cross entropy.

Our theoretical analyses_will_assume discrete distributions. However,_there is no loss of_generality in this assumption. Rigorous treatments_of probability_(measure theory)_treat integrals (either Reimann or Lebesque) as limits of increasingly fine_binnings. A continuous density can always_be viewed as a_limit of_discrete_distributions. Although our_proofs_are given_for discrete case, all our formal limitations_on the_measurement of mutual information apply to_continuous case as well._See_[@diffent] for a discussion of continuous_information theory. Additional comments on this_point are given in section \[sec:limitations2\].

The_Donsker-Varadhan_Lower_Bound
================================

Mutual information can be written_as a KL divergence. $$I(X,Y) =_KL(P_{X,Y},P_XP_Y)$$ Here $P_{X,Y}$_is a joint distribution on the