 diversity $\operatorname{\mathbb{ED}}$ achieves its maximal value $1$ if and only if the column vectors of $\tilde{M}_{\backslash y}$ are mutually orthogonal.

Note that most of the previous definitions of diversity, e.g., those defined on prediction errors, include the predictions $F_{y}^{k}$ on the true label $y$ [@liu1999ensemble; @liu1999simultaneous; @islam2003constructive; @kuncheva2003measures]. Since DNNs are no longer weak classifiers in the most cases [@goodfellow2016deep], encouraging $F_{y}^{k}$ to be diverse may make the loss on accuracy outweighs the gain for DNNs, or render the convergence point of the training process uncontrollable. In contrast, we define the diversity among the normalized non-maximal predictions $\tilde{F}^{k}_{\backslash y}$, which allows the maximal prediction of each network to be consistent with the true label, and thus will not affect ensemble accuracy. What’s more, promoting this diversity could improve robustness. Because, in the adversarial setting, when the maximal prediction corresponds to the true label $y$, the non-maximal predictions correspond to the labels $[L]\backslash\{y\}$, which include all potentially wrong labels returned for the adversarial examples. Thus a high diversity or inconsistency on the non-maximal predictions $F^{k}_{\backslash y}$ can lower down the transferability of adversarial examples among the networks, and further lead to better robustness of the ensemble.

The $K$ individual models $\{F^{k}(x,\theta^{k})\}_{k\in[K]}$; the training dataset $\mathcal{D}=\{(x_i, y_i)\}_{i\in [N]}$.\
Initialize each $\theta^k$ as $\theta^k_0$, the training step counters as $c_k=0$, $\varepsilon_k$ be the learning rate variables, $I=[K]$ be the indicator set, where $k\in[K]$. Calculate the objective on a mini-batch of data $\mathcal{D}_m$ $$\mathcal{L}_{\text{ADP}}^m=\frac{1}{\left|\mathcal{D}_m\right|} \sum_{(x_i, y_i) \in \mathcal{D}_m}\left[\mathcal{L}_{\text{ECE}}-\operatorname{\text{ADP}}_{\alpha,\
diversity $ \operatorname{\mathbb{ED}}$ achieves its maximal value $ 1 $ if and only if the column vectors of $ \tilde{M}_{\backslash y}$ are mutually extraneous. 

 notice that most of the former definitions of diversity, e.g., those define on prediction errors, include the prediction $ F_{y}^{k}$ on the on-key label $ y$   [ @liu1999ensemble; @liu1999simultaneous; @islam2003constructive; @kuncheva2003measures ]. Since DNNs are no longer weak classifier in the most cases   [ @goodfellow2016deep ], encouraging $ F_{y}^{k}$ to be divers may make the loss on accuracy outweighs the gain for DNNs, or render the overlap point of the training procedure uncontrollable. In contrast, we specify the diversity among the normalized non - maximal predictions $ \tilde{F}^{k}_{\backslash y}$, which allow the maximal prediction of each network to be consistent with the true label, and thus will not involve ensemble accuracy. What ’s more, promoting this diversity could improve robustness. Because, in the adversarial setting, when the maximal prediction corresponds to the genuine label $ y$, the non - maximal predictions correspond to the labels $ [ L]\backslash\{y\}$, which include all potentially wrong labels returned for the adversarial examples. Thus a high diversity or inconsistency on the non - maximal predictions $ F^{k}_{\backslash y}$ can lower down the transferability of adversarial examples among the networks, and further moderate to better robustness of the ensemble. 

 The $ K$ individual mannequin $ \{F^{k}(x,\theta^{k})\}_{k\in[K]}$; the education dataset $ \mathcal{D}=\{(x_i, y_i)\}_{i\in [ N]}$.\ 
 format each $ \theta^k$ as $ \theta^k_0 $, the training step counters as $ c_k=0 $, $ \varepsilon_k$ be the learning rate variables, $ I=[K]$ be the indicator set, where $ k\in[K]$. count the objective on a mini - batch of data $ \mathcal{D}_m$ $ $ \mathcal{L}_{\text{ADP}}^m=\frac{1}{\left|\mathcal{D}_m\right| } \sum_{(x_i, y_i) \in \mathcal{D}_m}\left[\mathcal{L}_{\text{ECE}}-\operatorname{\text{ADP}}_{\alpha,\
 digersity $\operatorname{\mathnb{ED}}$ achieves its maximel valus $1$ if ana only if the column vectors od $\tilee{M}_{\backslash y}$ are mutjally ortjogonal.

Nite uhat most of the 'devious definjbions if diversity, e.n., those defhned on predicdiun errors, include the predictions $F_{y}^{k}$ on the tgue label $y$ [@liu1999gnsemnje; @ljl1999slmultaneous; @islam2003constructive; @kuhcheva2003mtasures]. Since DNNs are no longer weak classivierd in the most cased [@goodfellow2016eeep], wncouraging $W_{y}^{k}$ to be diverse may jake the loss on accuracy outwekghs che gain fot SNJv, or render the bonvergence point of dhe traoning process mnconvroloable. In contrast, we vefine the diversity among tha uormalized non-maximal peedicjions $\tilaw{F}^{k}_{\caciskaah y}$, wjici allows ths maximal peediction of each nttwjgl to be consjstent wyth the true label, and thus will not afxecf ensemble accuracy. Whar’s more, promoting thid diversiey could improve robustness. Because, in the adversdrial retuikn, whdb hhe maximal prediction corresponds to the truq lsbvl $y$, the non-maximcl predictions cprgexkond to the lacels $[L]\yzcislash\{y\}$, which inclkde all potebtially wwong labels returned for the adcersarial excmpoes. Thus a high dirersity or iuconsixtencu on the non-maximal prebictiohs $F^{k}_{\backslwsh y}$ can uower down the tfanxfarability of adversarial evamples anong the negworls, and further lfad to better robustness of tke envemble.

The $N$ individual models $\{F^{k}(x,\theta^{k})\}_{k\ii[L]}$; the trainimg danaset $\matkcal{D}=\{(x_l, y_i)\}_{i\in [N]}$.\
Initiajize each $\thetc^k$ as $\thzta^k_0$, tfe traininf step rounters as $s_k=0$, $\varepsilon_n$ be the leariing rate varuablws, $I=[K]$ bd the indicator set, wherv $h\in[K]$. Calculate the objcctivg kn a mini-batch if eata $\mathcal{D}_m$ $$\katfcaj{L}_{\nexv{ADP}}^m=\ssac{1}{\left|\mathcdl{D}_m\figfy|} \sum_{(b_i, y_i) \in \mabhcxl{D}_m}\keft[\mathcal{L}_{\text{ECE}}-\o[eraforname{\text{ADP}}_{\alphs,\
diversity $\operatorname{\mathbb{ED}}$ achieves its maximal value $1$ only the column of $\tilde{M}_{\backslash y}$ most the previous definitions diversity, e.g., those on prediction errors, include the predictions on the true label $y$ [@liu1999ensemble; @liu1999simultaneous; @islam2003constructive; @kuncheva2003measures]. Since DNNs are no weak classifiers in the most cases [@goodfellow2016deep], encouraging $F_{y}^{k}$ to be diverse may the on outweighs gain for DNNs, or render the convergence point of the training process uncontrollable. In contrast, we the diversity among the normalized non-maximal predictions $\tilde{F}^{k}_{\backslash which allows the maximal of each network to be with true label, thus not ensemble accuracy. What’s promoting this diversity could improve robustness. Because, in the adversarial setting, when the maximal prediction corresponds to true label non-maximal predictions to labels which include all labels returned for the adversarial examples. diversity or inconsistency on the non-maximal predictions $F^{k}_{\backslash can lower the transferability of adversarial examples among networks, and further lead to better robustness of ensemble. The $K$ individual models $\{F^{k}(x,\theta^{k})\}_{k\in[K]}$; the training dataset $\mathcal{D}=\{(x_i, y_i)\}_{i\in [N]}$.\ Initialize each $\theta^k$ the training step counters $c_k=0$, $\varepsilon_k$ be learning variables, be indicator set, $k\in[K]$. Calculate the objective on a mini-batch of data $\mathcal{D}_m$ $$\mathcal{L}_{\text{ADP}}^m=\frac{1}{\left|\mathcal{D}_m\right|} y_i) \in \mathcal{D}_m}\left[\mathcal{L}_{\text{ECE}}-\operatorname{\text{ADP}}_{\alpha,\
 diversity $\operatorname{\mathBb{ED}}$ achievEs its MaxImaL vAlue $1$ If anD only if the coluMN vecTors of $\tilde{M}_{\backslash y}$ Are muTuALly oRThOgonaL.

Note thAT mOST of ThE pRevIoUS dEfiniTioNs of divErsity, e.g., thOse DeFined on prediCTiOn errors, inCluDe the predictIonS $F_{y}^{k}$ on ThE trUE labeL $y$ [@lIu1999ensEmble; @lIU1999simulTaneous; @isLaM2003ConstrUCtive; @kuNCHeVa2003meAsures]. Since DNNs arE No LOnger weak classIfiers In THe MOSt cAseS [@goodfelloW2016dEep], enCOuraginG $f_{y}^{K}$ TO Be dIVerse may make tHe loss on accURacY outweIgHs tHE gain fOr DNNS, oR RenDer the conveRgenCe point of The traINing proCEss uncoNtrollAblE. In ContRAsT, wE deFiNE thE DiVerSIty Among the NoRmAlizeD non-MAXIMal pRedIctiOns $\tiLde{F}^{k}_{\backslasH y}$, wHich ALloWs the MaximAl prEdIctioN of eacH netwOrK to be consistent With The true laBel, AnD thUs Will nOT affecT enSemBle accuRacy. WhaT’S moRe, PROMoTing this diversity cOuLD ImProve robUstnesS. beCaUSe, in the aDvErsAriaL SEttinG, wheN ThE maximal PredicTIoN cOrrespoNdS to the TrUe lAbeL $y$, the NOn-maXimal pRedictioNs corREspond to the labELs $[L]\backslash\{y\}$, WHiCH InCLude All Potentially WronG LabeLs reTUrNed FOr the AdverSaRIaL Examples. Thus a high diVeRsity oR incoNsistency on thE non-maximaL PREdictionS $F^{k}_{\bACkSLash y}$ can lower dOwn thE transferaBIlity of aDversArial exaMples amonG THe networKs, aNd fUrtHer LEAd To better robusTNEss oF tHe ensemBle.

the $K$ indIviDuaL moDelS $\{F^{K}(x,\theta^{k})\}_{k\In[K]}$; the trAiNiNg DaTasEt $\matHCal{D}=\{(x_i, y_i)\}_{I\iN [N]}$.\
INiTiaLize eACh $\thetA^k$ as $\tHeta^K_0$, tHe TRaiNing steP CoUNTers As $C_k=0$, $\VarePsiLoN_k$ be tHe leARniNg rate vAriables, $I=[k]$ be THe inDiCaTor set, wHere $k\in[K]$. CalcuLaTe the objecTiVe oN a mini-BATch of datA $\mathcal{D}_m$ $$\mathcal{L}_{\text{Adp}}^m=\frac{1}{\lEft|\MathcAl{D}_m\Right|} \sum_{(x_I, y_i) \In \mathCal{d}_M}\left[\mAthcal{l}_{\text{eCe}}-\opERAtornAME{\tExt{aDp}}_{\alpha,\
 diversity $\operatorname{ \mathbb{ED }}$ a chi eve sitsmaxi mal value $1$i f an d only if the column v ector so f $\ t il de{M} _{\back s la s h  y} $ar e m ut u al ly or tho gonal.
Note that  mo st  of the prev i ou s definiti ons  of diversit y,e.g.,th ose defin edon pr edicti o n erro rs, inclu de the pr e diction s $F _{y} ^{k}$ on the true la b el $y$ [@liu19 99ense mb l e; @ liu 199 9simultane ou s; @i s lam2003 c on s t r uct i ve; @kuncheva 2003measure s ].SinceDN Nsa re nolonge rw eak  classifier s in  the most  cases [@goodf e llow201 6deep] , e nco urag i ng  $ F_{ y} ^ {k} $  t o b e  di verse ma yma ke th e lo s s o n ac cur acyoutwe ighs the gain  fo r DN N s,or re ndertheco nverg ence p ointof  the training p roce ss uncont rol la ble .In co n trast,  we  de fine th e diver s ity  a m o n gthe normalized non -m a x im al predi ctions $\ ti l de{F}^{k }_ {\b acks l a sh y} $, w h ic h allows  the m a xi ma l predi ct ion of  e ach  ne twork to b e cons istent w ith t h e true label,a nd thus willn ot a ff e ct e nse mble accura cy.W hat’ s mo r e,  pr o motin g thi sd iv e rsity could improve  r obustn ess.Because, in t he adversa r i a l settin g, w h en the maximal pr edict ion corres p onds tothe t rue labe l $y$, th e non-maxi mal  pr edi cti o n scorrespond to t he l ab els $[L ]\b ackslas h\{ y\} $,whi ch  includeall pote nt ia ll ywro ng la b els retu rn edfo r t he ad v ersari al ex ampl es .T hus  a high di v e rsit yor  inc ons is tency  ont henon-max imal pred ict i ons$F ^{ k}_{\ba ckslash y}$ c an  lower dow nthe  trans f e rability  of adversarial example s  amongthe  netw orks , and fur the r lead  to better  robus tness  o f t h e  ense m b le .
Th e $K$ indi v i dua l mod el s $\ {F^{k}( x,\theta^{k})\}_{k \ in[ K]}$; the tra ini ng d a t as et$ \m a thc al { D}= \ { (x_i, y_i)\}_{i \in [N]}$. \I ni tialize ea c h $ \t heta^k$  as $\t heta^ k _0$, th e trainin g step co un ters a s $ c_k=0$, $\ varepsil on_k$ bet he le a rn ing r ate  varia bl es,  $I=[ K]$ be the  indi catorse t, whe re $k \i n[K]$. C alculate the objectiveon a m ini-b atc h of data  $\ m ath cal{D}_m$  $$\ mathcal{L} _{\ tex t{ADP }}^ m =\fra c{1} { \l eft | \math cal{ D }_m\right | }\su m _ {( x_i, y_i) \ i n \ma thcal {D} _ m}\lef t[\m athcal{L}_{\text{ E CE}}-\operator name { \ tex t{A D P}}_ {\ alpha,\
 diversity_$\operatorname{\mathbb{ED}}$ achieves_its maximal value $1$_if and_only_if the_column_vectors of $\tilde{M}_{\backslash_y}$ are mutually_orthogonal.

Note that most of_the previous definitions_of_diversity, e.g., those defined on prediction errors, include the predictions $F_{y}^{k}$ on the true_label_$y$ [@liu1999ensemble; @liu1999simultaneous;_@islam2003constructive;_@kuncheva2003measures]._Since DNNs are no longer_weak classifiers in the most_cases [@goodfellow2016deep], encouraging_$F_{y}^{k}$ to be diverse may make the loss_on_accuracy outweighs the_gain for DNNs, or render the convergence point of_the training process uncontrollable. In contrast,_we define the_diversity_among_the normalized non-maximal predictions_$\tilde{F}^{k}_{\backslash y}$, which allows the maximal_prediction of each network to be_consistent with the true label, and thus_will not affect ensemble accuracy. What’s_more, promoting this diversity could_improve robustness._Because, in the adversarial setting,_when the maximal_prediction corresponds_to the true_label $y$, the non-maximal predictions correspond_to the labels_$[L]\backslash\{y\}$, which include all potentially wrong_labels_returned for the_adversarial_examples._Thus a_high diversity or_inconsistency_on the_non-maximal_predictions $F^{k}_{\backslash y}$ can lower down_the_transferability of adversarial examples among the networks,_and further lead to_better_robustness of the ensemble.

The_$K$ individual models $\{F^{k}(x,\theta^{k})\}_{k\in[K]}$; the_training dataset $\mathcal{D}=\{(x_i, y_i)\}_{i\in [N]}$.\
Initialize each_$\theta^k$ as_$\theta^k_0$, the_training step counters as $c_k=0$, $\varepsilon_k$ be the learning rate variables,_$I=[K]$ be the indicator set, where_$k\in[K]$. Calculate the objective_on a_mini-batch_of data $\mathcal{D}_m$_$$\mathcal{L}_{\text{ADP}}^m=\frac{1}{\left|\mathcal{D}_m\right|}_\sum_{(x_i, y_i)_\in \mathcal{D}_m}\left[\mathcal{L}_{\text{ECE}}-\operatorname{\text{ADP}}_{\alpha,\