LeNet [@SzegedyLJSRAEVR15] and ResNet [@HeZRS16]. Besides the success on image categorization, features extracted from the last few layers are applied for generic tasks. [@DonahueJVHZTD14] adopts the deep features from the last two layers in AlexNet and shows the impressive performance on visual recognition with different applications. After that, [@QianJZL15] applies deep features for distance metric learning and achieves the overwhelming performance to the hand-crafted features on fine-grained visual categorization. [@mormont2018comparison] compares deep features from different neural networks and ResNet shows the best results. Besides the model pre-trained on ImageNet, [@ZhouLXTO14] proposes to learn deep features with the large-scale scene data set to improve the performance on the scene recognition task. All of these work directly extract features from the model learned with ERM as the objective. In contrast, we develop an algorithm that is tailored to learn robust deep representations. Note that deep features can be extracted from multiple layers of deep models and we focus on the layer before the final fully-connected layer in this work.

Recently, distributionally robust optimization that aims to optimize the worst-case performance has attracted much attention [@ChenLSS17; @NamkoongD16; @sinha2018certifiable]. [@NamkoongD16] proposes to optimize the performance with worst-case distribution over examples that is derived from the empirical distributions. [@ChenLSS17] extends the problem to the non-convex loss function, but they require a near-optimal oracle for the non-convex problem to learn the robust model. [@sinha2018certifiable] introduces the adversarial perturbation on each example for robustness. Most of these algorithms only consider the example-level robustness. In contrast, we propose the hierarchically robust optimization, that considers example-level and concept-level robustness simultaneously, to learn the representations for real applications.

Hierarchical Robustness
=======================

Problem Formulation {#sec:method}
-------------------

Let $\x_i$ denote an image and $y_i\in \R$ be its corresponding label. Given a benchmark data set $\{\x_i, y_i\}$ where $i=1,\ldots, N$, the parameters $\theta$ in a deep neural network can be learned by solving the optimization problem $$\begin{aligned}
\
LeNet   [ @SzegedyLJSRAEVR15 ] and ResNet   [ @HeZRS16 ]. Besides the success on image categorization, features educe from the final few layers are applied for generic tasks. [ @DonahueJVHZTD14 ] adopt the deep features from the concluding two layers in AlexNet and shows the impressive operation on ocular recognition with unlike applications. After that, [ @QianJZL15 ] applies deep features for distance measured learning and achieves the overpowering performance to the hand - craft features on fine - grained ocular categorization. [ @mormont2018comparison ] compares bass features from unlike neural network and ResNet shows the best results. Besides the exemplar pre - trained on ImageNet, [ @ZhouLXTO14 ] proposes to learn deep features with the large - scale scene data set to improve the performance on the scene recognition task. All of these work directly educe features from the model memorize with ERM as the aim. In contrast, we develop an algorithm that is tailored to learn robust deep representations. Note that cryptic features can be extracted from multiple layers of deep models and we focus on the level before the final fully - connected layer in this work. 

 Recently, distributionally robust optimization that aims to optimize the bad - case performance has attracted much attention   [ @ChenLSS17; @NamkoongD16; @sinha2018certifiable ]. [ @NamkoongD16 ] proposes to optimize the operation with worst - case distribution over examples that is derived from the empirical distributions. [ @ChenLSS17 ] extends the problem to the non - convex loss routine, but they require a dear - optimal oracle for the non - convex trouble to learn the robust model. [ @sinha2018certifiable ] introduces the adversarial disturbance on each example for robustness. Most of these algorithm only consider the example - horizontal surface robustness. In contrast, we project the hierarchically robust optimization, that consider example - level and concept - level robustness simultaneously, to memorize the representations for real applications. 

 Hierarchical Robustness 
 = = = = = = = = = = = = = = = = = = = = = = = 

 Problem Formulation { # sec: method } 
 ------------------- 

 permit $ \x_i$ denote an image and $ y_i\in \R$ be its corresponding label. Given a benchmark data set $ \{\x_i, y_i\}$ where $ i=1,\ldots, N$, the parameter $ \theta$ in a deep neural network can be learned by solving the optimization problem $ $ \begin{aligned } 
 \
LeNft [@SzegedyLJSRAEVR15] and RerNet [@HeZRS16]. Besidgs the snccess kn image categorization, features extcactwd frim the last few layers are applped for gwnermc tasks. [@DonahueOBHZTD14] adopts fme dezp features from the last dwo layers in DldxUet and shows the impressive performwnce on vlsual recognityon eyth spfnerent applications. After that, [@QjanJZL15] epplies deep festures for distance metric leagning and achieves the overwhgmmigt performancd to the hand-crafted fgatures on fine-grained visual cagegornzation. [@mornobt2018clkparison] conparef deep featuvvs from differrnt neural neteorns qnd ResNet shows the uest results. Besides the moden 're-trained on ImageNer, [@ZhouLFTO14] [ropuwes to lxarh deep feetures with the large-sxale scene data set tj improve the lerforiagce on the scene recognition task. All ox tgese work directly extrqct features from the model lewrned with ERM as the objective. In contrast, we deeelop xn cogorigym that is tailored to learn robust deep represqhtstpons. Note that decp features can be edttwcted from muutiple lagers of deep modeld and wg focuw on the jayet before the final fully-conbected layer un this work.

Recentpy, distribucionalky ronust optimization that cims tk optimize hhe worst-dxse performance fas adtracted much attention [@ChegLSS17; @NamkiongB16; @sinha2018cdrtigiable]. [@NamkoongD16] proposes to optimize thf peryormatce with wlrst-case distribution over exam'kes that is cesivvd from tke emplrical distribueions. [@ChenLSS17] gxtends tke proclem to thv non-convxx loss funceion, but they tequire a neac-optimal jracoe fir the vun-convex problrm to leagn the robust model. [@sinhe2018cerjiriable] introduczr the adversariak pdrttrhavion jt each exampne fur fpbustvess. Most on tfese algorithms only convided the example-level rjbustnesw. In conerast, we proppse the hierarchicwlly cobust optikizwtion, that considers example-lebel and clnccpt-level robuftnews simultanepusly, to learn the representations for ceal applications.

Hieratchical Robustness
=======================

Proyltm Formulatioi {#sec:mqthod}
-------------------

Let $\f_i$ denote an image abd $y_i\in \R$ be its gorresponding label. Giben a tenchlark data set $\{\x_i, y_i\}$ where $i=1,\ldots, N$, the parameters $\theta$ in a deep neural betwork can be leadned by smlriny the o[timmzetion problem $$\begpn{aligned}
\
LeNet [@SzegedyLJSRAEVR15] and ResNet [@HeZRS16]. Besides the image features extracted the last few tasks. adopts the deep from the last layers in AlexNet and shows the performance on visual recognition with different applications. After that, [@QianJZL15] applies deep features distance metric learning and achieves the overwhelming performance to the hand-crafted features on visual [@mormont2018comparison] deep from different neural networks and ResNet shows the best results. Besides the model pre-trained on ImageNet, proposes to learn deep features with the large-scale data set to improve performance on the scene recognition All these work extract from model learned with as the objective. In contrast, we develop an algorithm that is tailored to learn robust deep representations. that deep be extracted multiple of models and we the layer before the final fully-connected work. Recently, distributionally robust optimization that aims to the worst-case has attracted much attention [@ChenLSS17; @NamkoongD16; [@NamkoongD16] proposes to optimize the performance with worst-case over examples that is derived from the empirical distributions. [@ChenLSS17] extends the problem to the function, but they require near-optimal oracle for non-convex to the model. [@sinha2018certifiable] the adversarial perturbation on each example for robustness. Most of these only consider the example-level robustness. In contrast, we propose the optimization, considers example-level and robustness simultaneously, to learn representations real applications. Hierarchical Robustness Formulation ------------------- an and \R$ be its corresponding Given a benchmark data set y_i\}$ where $i=1,\ldots, N$, deep neural network can be learned by solving optimization problem $$\begin{aligned} \
LeNet [@SzegedyLJSRAEVR15] and ReSNet [@HeZRS16]. BEsideS thE suCcEss oN imaGe categorizatiON, feaTures extracted from the lAst feW lAYers ARe AppliEd for geNErIC TasKs. [@doNahUejvHzTD14] adOptS the deeP features fRom ThE last two layeRS iN AlexNet anD shOws the impresSivE perfoRmAncE On visUal RecogNition WIth difFerent appLiCAtions. aFter thaT, [@qIaNJZL15] Applies deep featurES fOR distance metriC learnInG AnD AChiEveS the overwhElMing pERformanCE tO THE haND-crafted featuRes on fine-grAIneD visuaL cAteGOrizatIon. [@moRmONt2018cOmparison] coMparEs deep feaTures fROm diffeREnt neurAl netwOrkS anD ResnEt ShOws ThE BesT ReSulTS. BeSides the MoDeL pre-tRainED ON imagENeT, [@ZhoULXTO14] Proposes to leaRn dEep fEAtuRes wiTh the LargE-sCale sCene daTa set To Improve the perfoRmanCe on the scEne ReCogNiTion tASk. All oF thEse Work dirEctly exTRacT fEATUrEs from the model learNeD WItH ERM as thE objecTIvE. IN Contrast, We DevElop AN AlgorIthm THaT is tailoRed to lEArN rObust deEp RepresEnTatIonS. Note THat dEep feaTures can Be extRActed from multiPLe layers of deeP MoDELs ANd we FocUs on the layeR befORe thE finAL fUllY-ConneCted lAyER iN This work.

Recently, disTrIbutioNally Robust optimizAtion that aIMS To optimiZe thE WoRSt-case performaNce haS attracted MUch attenTion [@CHenLSS17; @NaMkoongD16; @siNHA2018certifiAblE]. [@NaMkoOngd16] PRoPoses to optimiZE The pErFormancE wiTh worst-CasE diStrIbuTiOn over exaMples thaT iS dErIvEd fRom thE EmpiricaL dIstRiButIons. [@CHEnLSS17] eXtendS the PrObLEm tO the non-COnVEX losS fUnCtioN, buT tHey reQuirE A neAr-optimAl oracle fOr tHE non-CoNvEx problEm to learn the rObUst model. [@siNhA2018ceRtifiaBLE] introduCes the adversarial perturBAtion on EacH examPle fOr robustnEss. most of TheSE algorIthms oNly coNsIdeR THe exaMPLe-LevEl Robustness. iN ConTrast, We PropOse the hIerarchically robusT OptImization, that ConSideRS ExAmpLE-lEVel AnD ConCEPt-level robustneSs simultanEoUSlY, to learn thE RepReSentatiOns for rEal apPLicatioNs.

HierarcHical RobuStNess
=======================

pROblEm FormulatIon {#sec:meThod}
-------------------

Let $\x_i$ DEnote AN iMage aNd $y_I\in \R$ be ItS coRrespOnding LAbeL. GiveN a bencHmArk datA set $\{\x_I, y_I\}$ where $i=1,\lDots, N$, the parameters $\theta$ In a deeP neurAl nEtwork can Be lEArnEd by solviNg thE optimizatIon ProBlem $$\bEgiN{AlignEd}
\
LeNet [@SzegedyLJSRAEVR15]  and ResNe t [@H eZR S16 ].  Bes ides  the success o n  ima ge categorization, fea tures  e x trac t ed  from  the la s tf e w l ay er s a re ap plied  fo r gener ic tasks.[@D on ahueJVHZTD14 ]  a dopts thedee p features f rom  the l as t t w o lay ers  in A lexNet and sh ows the i mp r essive perform a n ce  onvisual recognitio n  w i th different a pplica ti o ns . Aft erthat, [@Qi an JZL15 ]  applie s  d e e p  fe a tures for dis tance metri c  le arning  a nda chieve s the  o v erw helming per form ance to t he han d -crafte d  featur es onfin e-g rain e dvi sua lc ate g or iza t ion . [@morm on t2 018co mpar i s o n ] co mpa resdeepfeatures from  di ffer e ntneura l net work sand R esNetshows  t he best results . Be sides the  mo de l p re -trai n ed onIma geN et, [@Z houLXTO 1 4]pr o p o se s to learn deep fe at u r es  with th e larg e -s ca l e sceneda tasett o  impr ovet he  perform ance o n  t he  scenere cognit io n t ask . All of t hese w ork dire ctlye xtract feature s  from the mod e ll e ar n ed w ith  ERM as the  obj e ctiv e. I n  c ont r ast,we de ve l op an algorithm that i stailor ed to  learn robust  deep repr e s e ntations . No t et hat deep featu res c an be extr a cted fro m mul tiple la yers of d e e p models  an d w e f ocu s on  the layer be f o re t he  finalful ly-conn ect edlay erin  this wor k.

Rece nt ly ,di str ibuti o nally ro bu stop tim izati o n that  aims  toop ti m ize  the wo r st - c asepe rf orma nce  h as at trac t edmuch at tention [ @Ch e nLSS 17 ;@Namkoo ngD16; @sinha 20 18certifia bl e].  [@Nam k o ongD16]proposes to optimize th e  perfor man ce wi th w orst-case  di stribu tio n  overexampl es th at  is d erive d fr omth e empirica l dis tribu ti ons.  [@Chen LSS17] extends the pro blem to the n on- conv e x  l oss fu n cti on ,  bu t they require anear-optim al or acle for t h e n on -convex  proble m tol earn th e robustmodel. [@ si nha2 0 1 8ce rtifiable]  introdu ces the a d versa r ia l per tur bation  o n e ach e xample for  robu stness .Most o f the se  algorit hms only consider the e xample -leve l r obustness . I n  co ntrast, w e pr opose thehie rar chica lly robus t op t im iza t ion,that considers ex amp l e -l evel and co n c e pt- level  ro b ustnes s si multaneously, tol earn the repre sent a t ion s f o r re al  applications. 

H ie r a rchicalRo bustness
== ======== == = ===== =====
Probl em Form u l at i on {#s ec:m eth od}
----- --- -- - ------- -
L e t $\x_ i$ d en ote an  image and$ y _i\in \R$ be its  corr e s pondi n g l abel.  G iven ab ench mark dataset $\{\x_i , y_i\ }$ w here$i=1,\l do ts, N$ , t he  parameter s  $\theta$  in a  deep n eu ralnet work c an b e learn ed b ysol ving theo p ti m iz at i onprob lem $ $\ begi n{aligned } 
\
LeNet [@SzegedyLJSRAEVR15] and_ResNet [@HeZRS16]. Besides_the success on image_categorization, features_extracted_from the_last_few layers are_applied for generic_tasks. [@DonahueJVHZTD14] adopts the_deep features from_the_last two layers in AlexNet and shows the impressive performance on visual recognition with_different_applications. After_that,_[@QianJZL15]_applies deep features for distance_metric learning and achieves the_overwhelming performance_to the hand-crafted features on fine-grained visual categorization._[@mormont2018comparison]_compares deep features_from different neural networks and ResNet shows the best_results. Besides the model pre-trained on_ImageNet, [@ZhouLXTO14] proposes_to_learn_deep features with the_large-scale scene data set to improve_the performance on the scene recognition_task. All of these work directly extract_features from the model learned with_ERM as the objective. In_contrast, we_develop an algorithm that is_tailored to learn_robust deep_representations. Note that_deep features can be extracted from_multiple layers of_deep models and we focus on_the_layer before the_final_fully-connected_layer in_this work.

Recently, distributionally_robust_optimization that_aims_to optimize the worst-case performance has_attracted_much attention [@ChenLSS17; @NamkoongD16; @sinha2018certifiable]. [@NamkoongD16] proposes to_optimize the performance with_worst-case_distribution over examples that_is derived from the empirical_distributions. [@ChenLSS17] extends the problem to_the non-convex_loss function,_but they require a near-optimal oracle for the non-convex problem to_learn the robust model. [@sinha2018certifiable] introduces_the adversarial perturbation on_each example_for_robustness. Most of_these_algorithms only_consider the example-level robustness. In contrast, we_propose the_hierarchically robust optimization, that considers example-level_and concept-level robustness simultaneously,_to_learn the representations for real applications.

Hierarchical_Robustness
=======================

Problem Formulation {#sec:method}
-------------------

Let $\x_i$ denote an_image and $y_i\in \R$ be_its_corresponding_label. Given a benchmark data_set $\{\x_i, y_i\}$ where $i=1,\ldots, N$,_the parameters $\theta$_in a deep neural network can be_learned_by solving the optimization problem $$\begin{aligned}
\