paulus:2017:arxiv], there is still no guarantee that the generated summaries are grammatical and convey the same meaning as the original document does. It seems that extractive models are more reliable than their abstractive counterparts.

However, extractive models require sentence level labels, which are usually not included in most summarization datasets (most datasets only contain document-summary pairs). Sentence labels are usually obtained by rule-based methods (e.g., maximizing the ROUGE score between a set of sentences and reference summaries) and may not be accurate. Extractive models proposed recently [@cheng:2016:acl; @nallapati:2017:aaai] employ hierarchical document encoders and even have neural decoders, which are complex. Training such complex neural models with *inaccurate* *binary* labels is challenging. We observed in our initial experiments on one of our dataset that our extractive model (see Section \[sec:sum\] for details) overfits to the training set quickly after the second epoch, which indicates the training set may not be fully utilized. Inspired by the recent pre-training work in natural language processing [@peters:2018:naacl; @radford:2018:nips; @devlin:2018:arxiv], our solution to this problem is to first pre-train the “complex”’ part (i.e., the hierarchical encoder) of the extractive model on unlabeled data and then we learn to classify sentences with our model initialized from the pre-trained encoder. In this paper, we propose, which stands for [**HI**]{}erachical [**B**]{}idirectional [**E**]{}ncoder [**R**]{}epresentations from [**T**]{}ransformers. We design an unsupervised method to pre-train for document modeling. We apply the pre-trained to the task of document summarization and achieve state-of-the-art performance on both the CNN/Dailymail and New York Times dataset.

Related Work
============

In this section, we introduce work on extractive summarization, abstractive summarization and pre-trained natural language processing models. For a more comprehensive review of summarization, we refer the interested readers to and.

#### Extractive Summarization

Extractive summarization aims to select important sentences (sometimes other textual units such as elementary discourse units (EDUs)) from a document as its summary. It is usually modeled as a sentence ranking problem by using
paulus:2017: arxiv ], there is still no guarantee that the generated summaries are grammatical and bring the like meaning as the original document does. It seem that extractive model are more reliable than their abstractive counterparts. 

 However, extractive exemplar require sentence degree labels, which are usually not admit in most summarization datasets (most datasets only contain text file - summary pairs). Sentence labels are usually obtained by rule - based method (e.g., maximizing the ROUGE score between a hardening of sentences and reference summaries) and may not be accurate. Extractive model proposed recently [ @cheng:2016: acl; @nallapati:2017: aaai ] employ hierarchical document encoders and even have neural decoder, which are complex. Training such complex neural models with * inaccurate * * binary * labels is challenging. We observed in our initial experiments on one of our dataset that our extractive model (see Section \[sec: sum\ ] for details) overfits to the training set quickly after the second epoch, which indicates the training set may not be amply utilized. Inspired by the late pre - training workplace in lifelike language processing [ @peters:2018: naacl; @radford:2018: nips; @devlin:2018: arxiv ], our solution to this problem is to first pre - train the “ complex ” ’ part (i.e., the hierarchical encoder) of the extractive model on unlabeled data and then we determine to classify sentences with our model initialized from the pre - trained encoder. In this paper, we aim, which stands for [ * * HI**]{}erachical [ * * B**]{}idirectional [ * * E**]{}ncoder [ * * R**]{}epresentations from [ * * T**]{}ransformers. We design an unsupervised method to pre - train for text file modeling. We give the pre - coach to the task of document summarization and achieve state - of - the - artwork performance on both the CNN / Dailymail and New York Times dataset. 

 Related Work 
 = = = = = = = = = = = = 

 In this section, we introduce work on extractive summarization, abstractive summarization and pre - trained natural language processing models. For a more comprehensive review of summarization, we consult the interested readers to and. 

 # # # # Extractive Summarization 

 Extractive summarization aim to select significant sentences (sometimes early textual units such as elementary discourse unit (EDUs) ) from a document as its summary. It is usually model as a sentence rank problem by use
paupus:2017:arxiv], there is still ko guarantee thaj rhe geierated summarids are grammatical and conveb thw samt meaning as the orkginal dobument dows. Iu seems that extredtive models zve moxe reliable than their absdractive countaroaxts.

However, extractive models require sentenve level labels, rhicn are lsmally not included in most summadizatioi datasets (most datasets only contain dockmenh-summary pairs). Senhence labelw arq usually obtxined by rlne-based mejhods (e.g., maximizing the ROUGE scure bztween a sej kf ventences aid refvrence summarlvs) and kay not be accurate. Ewtracvive models proposed receitly [@cheng:2016:acl; @nallapwti:2017:aaai] ekpmoy hierarchical eoxumenj encmderr ana ebei hzve nekram decoders, which are xomplex. Training suvh bpmplex neuram modejs with *inaccurate* *binary* labels is challtnginf. We observed in our inutial experiments on lne of ouw dataset that our extractive model (see Section \[sac:sum\] wor dctaiuw) lverfits to the training set quickly after thq stcokd epoch, which ikdicates the trainonh xgt may not be wully bfimized. Inspired by hhe recgnt prw-training worl in natural language procewsing [@peters:2018:uaaxl; @radford:2018:nips; @devpin:2018:arxiv], oux solujion tp this problem is to fixst prs-train the “fomplex”’ pzft (i.e., the hierarzhibal ancoder) of the extractive iodel on nnlabzled datx anc then we learn ho classify sentences wihh out moden initialixed from the pre-trained encoder. In this paper, wa pgopose, whnch stsnds for [**HI**]{}erwchical [**B**]{}idireetional [**Z**]{}ncodef [**R**]{}epresennations fcom [**T**]{}ransforiers. We desigt an unsupervmsed methjd ti prw-train wur document moceling. We apply the pre-trained to the tari of document sbonarization and schkevq dtete-of-eve-art perforkancd ov both the CNN/Daiohmaik and New York Times datzset.

Related Work
============

In tmis sectiin, we ineroduce work pn extractive summwrizavion, austracyivg summarization and pre-trained hatural lwngmage processigg midels. For a kore comprehensive review of summarizatmon, we refer the intergsted readers to and.

#### Gxtvactive Summerizatyon

Extracdive summarization aums to select impprtant sentences (somefimes mther textual units such as elementary discourse units (EDUs)) from a document as its snmiary. It is hsuakly mmdzleb as a fentxnre ranking problek by using
paulus:2017:arxiv], there is still no guarantee that summaries grammatical and the same meaning It that extractive models more reliable than abstractive counterparts. However, extractive models require level labels, which are usually not included in most summarization datasets (most datasets contain document-summary pairs). Sentence labels are usually obtained by rule-based methods (e.g., maximizing ROUGE between set sentences and reference summaries) and may not be accurate. Extractive models proposed recently [@cheng:2016:acl; @nallapati:2017:aaai] employ document encoders and even have neural decoders, which complex. Training such complex models with *inaccurate* *binary* labels challenging. observed in initial on of our dataset our extractive model (see Section \[sec:sum\] for details) overfits to the training set quickly after the second which indicates set may be utilized. by the recent in natural language processing [@peters:2018:naacl; @radford:2018:nips; to this problem is to first pre-train the part (i.e., hierarchical encoder) of the extractive model unlabeled data and then we learn to classify with our model initialized from the pre-trained encoder. In this paper, we propose, which stands [**B**]{}idirectional [**E**]{}ncoder [**R**]{}epresentations from We design an method pre-train document We apply pre-trained to the task of document summarization and achieve state-of-the-art performance both the CNN/Dailymail and New York Times dataset. Related Work this we introduce work extractive summarization, abstractive summarization pre-trained language processing models. For comprehensive of the readers and. #### Extractive Summarization summarization aims to select important (sometimes other textual units (EDUs)) from a document as its summary. It usually modeled as a sentence ranking problem using
paulus:2017:arxiv], there is still no Guarantee tHat thE geNerAtEd suMmarIes are grammatiCAl anD convey the same meaning aS the oRiGInal DOcUment Does. It sEEmS THat ExTrActIvE MoDels aRe mOre reliAble than thEir AbStractive couNTeRparts.

HoweVer, Extractive moDelS requiRe SenTEnce lEveL labeLs, whicH Are usuAlly not inClUDed in mOSt summaRIZaTion Datasets (most datasETs ONly contain docuMent-suMmARy PAIrs). senTence labelS aRe usuALly obtaINeD BY RulE-Based methods (e.G., maximizing THe RoUGE scOrE beTWeen a sEt of sEnTEncEs and refereNce sUmmaries) aNd may nOT be accuRAte. ExtrActive ModEls PropOSeD rEceNtLY [@chENg:2016:Acl; @NAllApati:2017:aaaI] eMpLoy hiErarCHICAl doCumEnt eNcodeRs and even have NeuRal dECodErs, whIch arE comPlEx. TraIning sUch coMpLex neural models With *InaccuratE* *biNaRy* lAbEls is CHallenGinG. We ObserveD in our iNItiAl EXPErIments on one of our daTaSET tHat our exTractiVE mOdEL (see SectIoN \[seC:sum\] FOR detaIls) oVErFits to thE trainINg SeT quicklY aFter thE sEcoNd ePoch, wHIch iNdicatEs the traIning SEt may not be fullY Utilized. InspiREd BY ThE ReceNt pRe-training wOrk iN NatuRal lANgUagE ProceSsing [@PeTErS:2018:Naacl; @radford:2018:nips; @devLiN:2018:arxiv], Our soLution to this pRoblem is to FIRSt pre-traIn thE “CoMPlex”’ part (i.e., the hIerarChical encoDEr) of the eXtracTive modeL on unlabeLED data and TheN we LeaRn tO CLaSsify sentenceS WIth oUr Model inItiAlized fRom The Pre-TraInEd encoder. in this paPeR, wE pRoPosE, whicH Stands foR [**Hi**]{}erAcHicAl [**B**]{}idIRectioNal [**E**]{}nCodeR [**R**]{}EpREseNtationS FrOM [**t**]{}ranSfOrMers. we dEsIgn an UnsuPErvIsed metHod to pre-tRaiN For dOcUmEnt modeLing. We apply thE pRe-trained tO tHe tAsk of dOCUment sumMarization and achieve staTE-of-the-aRt pErforMancE on both thE CNn/DailyMaiL And New york TiMes daTaSet.

rELated wORk
============

in tHiS section, we INTroDuce wOrK on eXtractiVe summarization, absTRacTive summarizaTioN and PRE-tRaiNEd NAtuRaL LanGUAge processing moDels. For a moRe COmPrehensive REviEw Of summaRizatioN, we reFEr the inTerested rEaders to aNd.

#### extrACTivE SummarizaTion

ExtrActive sumMArizaTIoN aims To sElect iMpOrtAnt seNtenceS (SomEtimeS other TeXtual uNits sUcH as elemeNtary discourse units (EDUs)) From a dOcumeNt aS its summaRy. IT Is uSually modEled As a sentencE raNkiNg proBleM By usiNg
paulus:2017:arxiv], thereis still n o gua ran tee  t hatthegenerated summ a ries  are grammatical and c onvey  t h e sa m emeani ng as t h eo r igi na ldoc um e nt  does . I t seems  that extr act iv e models are mo re reliabl e t han their ab str active  c oun t erpar ts. 

How ever,e xtract ive model sr equire sentenc e le vellabels, which are us u ally not inclu ded in  m o st s umm ari zation dat as ets ( m ost dat a se t s onl y  contain docu ment-summar y  pa irs).Se nte n ce lab els a re usu ally obtain ed b y rule-ba sed me t hods (e . g., max imizin g t heROUG E  s co rebe t wee n  a  se t  of  sentenc es  a nd re fere n c e summ ari es)and m ay not be acc ura te.E xtr activ e mod elspr opose d rece ntly[@ cheng:2016:acl;  @na llapati:2 017 :a aai ]emplo y  hiera rch ica l docum ent enc o der sa n d  e ven have neural de co d e rs , whichare co m pl ex .  Trainin gsuc h co m p lex n eura l  m odels wi th *in a cc ur ate* *b in ary* l ab els  is  chal l engi ng. We  observe d ino ur initial exp e riments on on e  o f ou r  dat ase t that ourextr a ctiv e mo d el  (s e e Sec tion\[ s ec : sum\] for details)ov erfits  to t he training s et quickly a f ter theseco n de poch, which in dicat es the tra i ning set  maynot be f ully util i z ed. Insp ire d b y t her e ce nt pre-traini n g  wor kin natu ral  langua gepro ces sin g[@peters: 2018:naa cl ;@r ad for d:201 8 :nips; @ de vli n: 201 8:arx i v], ou r sol utio nto thi s probl e mi s  tofi rs t pr e-t ra in th e “c o mpl ex”’ pa rt (i.e.,  th e  hie ra rc hical e ncoder) of th eextractive  m ode l on u n l abeled d ata and then we learn t o  classi fysente nces  with our  mo del in iti a lizedfrom t he pr e- tra i n ed en c o de r.In  this pape r ,  we  prop os e, w hich st ands for [**HI**]{ } era chical [**B** ]{} idir e c ti ona l  [ * *E* *] { }nc o d er [**R**]{}epr esentation sf ro m [**T**]{ } ran sf ormers.  We des ign a n  unsupe rvised me thod to p re -tra i n  fo r document  modelin g. We app l y the pr e-tra ine d to t he  ta sk of  docum e ntsumma rizati on  and a chiev estate-of -the-art performance on  boththe C NN/ Dailymail  an d  Ne w York Ti mesdataset.
Rel ate d Wor k
= = ===== ==== = 
Int his s ecti o n, we int r od uce w or k on extrac t i v e s ummar iza t ion, a bstr active summarizat i on and pre-tra ined n atu ral lang ua ge processingmod el s .  For a m or e comprehen sive rev ie w  of s ummari zation , we re f e rt he int eres ted  readerstoan d .

####  E xt r active  Sum ma rizati on

Ex t ract i v e summarizationaimst o  sele c t i mport an t sente n ces(sometimes  other text ual un itssuchas elem en tary d isc ou rse units( EDUs)) fr om adocumen tas i tssummar y. I t is us uall ymod eled as a s en t en ce ran king  prob le m by  using
paulus:2017:arxiv], there_is still_no guarantee that the_generated summaries_are_grammatical and_convey_the same meaning_as the original_document does. It seems_that extractive models_are_more reliable than their abstractive counterparts.

However, extractive models require sentence level labels, which are_usually_not included_in_most_summarization datasets (most datasets only_contain document-summary pairs). Sentence labels_are usually_obtained by rule-based methods (e.g., maximizing the ROUGE_score_between a set_of sentences and reference summaries) and may not be_accurate. Extractive models proposed recently [@cheng:2016:acl;_@nallapati:2017:aaai] employ hierarchical_document_encoders_and even have neural_decoders, which are complex. Training such_complex neural models with *inaccurate* *binary*_labels is challenging. We observed in our_initial experiments on one of our_dataset that our extractive model_(see Section_\[sec:sum\] for details) overfits to_the training set_quickly after_the second epoch,_which indicates the training set may_not be fully_utilized. Inspired by the recent pre-training_work_in natural language_processing_[@peters:2018:naacl;_@radford:2018:nips; @devlin:2018:arxiv],_our solution to_this_problem is_to_first pre-train the “complex”’ part (i.e.,_the_hierarchical encoder) of the extractive model on_unlabeled data and then_we_learn to classify sentences_with our model initialized from_the pre-trained encoder. In this paper,_we propose,_which stands_for [**HI**]{}erachical [**B**]{}idirectional [**E**]{}ncoder [**R**]{}epresentations from [**T**]{}ransformers. We design an unsupervised_method to pre-train for document modeling._We apply the pre-trained_to the_task_of document summarization_and_achieve state-of-the-art_performance on both the CNN/Dailymail and New_York Times_dataset.

Related Work
============

In this section, we introduce_work on extractive summarization,_abstractive_summarization and pre-trained natural language processing_models. For a more comprehensive review_of summarization, we refer the_interested_readers_to and.

#### Extractive Summarization

Extractive summarization_aims to select important sentences (sometimes_other textual units_such as elementary discourse units (EDUs)) from_a_document as its summary. It is_usually_modeled as a sentence ranking problem_by_using