 one task to another. This ability to remember, learn and transfer information across tasks is referred to as continual learning [@Thrun1995; @Ruvolo2013; @Hassabis2017; @Parisi2019]. The major challenge for creating ML models with continual learning ability is that they are prone to *catastrophic forgetting* [@McClelland1995; @McCloskey1989; @Goodfellow2013; @French1999]. ML models tend to forget the knowledge learned from previous tasks when re-trained on new observations corresponding to a different (but related) task. Specifically when a deep neural network (DNN) is fed with a sequence of tasks, the ability to solve the first task will decline significantly after training on the following tasks. The typical structure of DNNs by design does not possess the capability of preserving previously learned knowledge without interference between tasks or catastrophic forgetting. In order to overcome catastrophic forgetting, a learning system is required to continuously acquire knowledge from the newly fed data as well as to prevent the training of the new data samples from destroying the existing knowledge.

In this paper, we propose a novel approach to continual learning with DNNs that addresses the catastrophic forgetting issue, namely a technique called *online leverage score sampling (OLSS)*. In OLSS, we progressively compress the input information learned thus far, along with the input from current task and form more efficiently condensed data samples. The compression technique is based on the statistical leverage scores measure, and it uses the concept of frequent directions in order to connect the series of compression steps for a sequence of tasks.

When thinking about continual learning, a major source of inspiration is the ability of biological brains to learn without destructive interference between older memories and generalize knowledge across multiple tasks. In this regard, the typical approach is enabling some form of episodic-memory in the network and consolidation [@McClelland1995] via replay of older training data. However, this is an expensive process and does not scale well for learning large number of tasks. As an alternative, taking inspiration from the neuro-computational models of complex synapses [@BennaFusi2016], recent work has focused on assigning some form of importance to parameters in a DNN and perform task-specific synaptic consolidation [@Kirkpatrick2017; @Zenke2017]. Here, we take a very different view of continual learning and find inspiration in the brains ability for dimensionality reduction [@Pang2016] to extract meaningful information from its environment and drive behavior. As such, we enable such
one task to another. This ability to remember, learn and transfer data across undertaking is referred to as continual learning [ @Thrun1995; @Ruvolo2013; @Hassabis2017; @Parisi2019 ]. The major challenge for create ML model with continual learning ability is that they are prone to * catastrophic forgetting * [ @McClelland1995; @McCloskey1989; @Goodfellow2013; @French1999 ]. ML exemplar tend to forget the knowledge learned from former undertaking when re - trained on modern observations corresponding to a unlike (but related) task. Specifically when a deep neural network (DNN) is fed with a sequence of tasks, the ability to resolve the first task will decline importantly after training on the following tasks. The distinctive structure of DNNs by purpose does not possess the capability of preserve previously learned knowledge without interference between undertaking or catastrophic forgetting. In order to overcome catastrophic forgetting, a learning system is required to continuously acquire knowledge from the newly fed data as well as to prevent the training of the new data samples from destroying the existing cognition. 

 In this composition, we propose a fresh overture to continual eruditeness with DNNs that addresses the catastrophic forgetting issue, namely a technique called * online leverage score sampling (OLSS) *. In OLSS, we progressively compress the stimulation information learned thus far, along with the input from current task and form more efficiently condense data samples. The compression technique is free-base on the statistical leverage scores measuring stick, and it uses the concept of frequent directions in order to connect the series of compaction steps for a sequence of tasks. 

 When thinking about continual learning, a major source of inspiration is the ability of biological brains to learn without destructive hindrance between older memory and generalize cognition across multiple tasks. In this regard, the typical approach is enable some form of episodic - memory in the network and consolidation [ @McClelland1995 ] via replay of previous training data. However, this is an expensive summons and does not scale well for learning big number of tasks. As an alternative, taking inspiration from the neuro - computational models of complex synapses [ @BennaFusi2016 ], recent work has concentrate on assigning some form of importance to parameter in a DNN and perform task - specific synaptic consolidation [ @Kirkpatrick2017; @Zenke2017 ]. Here, we assume a very different view of continual learning and receive inspiration in the brains ability for dimensionality reduction [ @Pang2016 ] to extract meaningful information from its environment and drive demeanor. As such, we enable such
 onf task to another. This anility to remembgr, learn and tdansfer knformation across tasks is cefeered uj as continual leafning [@Thrln1995; @Ruvolo2013; @Haswqbis2017; @Parism2019]. The mamjr cgwlleugx for creating KL models fith continual ldaxning ability is that they are prone to *catsshrophic forgetjing* [@KsClemland1995; @McCloskey1989; @Goodfellow2013; @French1999]. JL modens tend to fotget the knowledge learned frol previous tasks wjen re-traings og new observagions corrts'onding to z different (but related) task. Spdcifieally when q eeeo neural netxork (DGN) is fed wibn a sexuence pf tasks, the anilitb to solve the first task will decline signyficantly ayter training on the doolowitg tdsks. Rhe tykicel atructkre of DNNs bg design dows not possess the va[qbility of prsservigg previously learned knowledge without ittedference between tasks ir catastrophic forgejting. In owder to overcome catastrophic forgetting, a learnitg syagem if eeeuired to continuously acquire knowledge from fht nvwly fed data as cell as to prevemt tng training of jhe new dafa samples from dedtroyind the existing knoeledge.

In this paper, we propise a novel cppeoach to continual learning wnth DNMs thst addresses the catastxophic forgetting issue, najdly a technique zalkeg *online leverage score saipling (OLWS)*. Iu OLSS, wd prpgressyvely compgess bve input informatiln lecrned thus far, wlong with the input from curreiv task and fotm korv efficieutly cpndensed data samples. The cpmpressnon tezhnique is based mn the statystical leverdhe scores meesure, and it yses the covzept of frequemt directpous in ordwr to connect the xerkss of compressiin wteps for a seqienze jf tesks.

Wran thinking dboug cumtinuxl learning, x makor source of inspirdtioh is the ability og niologicao brains to learn witnout destructive ijterfxrence betwren older memories and generalize knowledgf agross multiplq tawks. In this xegard, the typical approach is enabling wome form of episoeic-memory in the nejwovk and consokidatyon [@McClenland1995] via replay of ilder training daua. However, this is an espensiee prlcess and does not scale well for learning large number of tasks. As an alrernatmvq, taking inapirstion fxom the nquro-rooputational modeks of complex synapses [@BennaFusm2016], recent wmrh has focused on assigning spmd form of impurtance to parameters in a DNN amd perform task-specific synaptiv consolidation [@Kirkpatrlck2017; @Zenke2017]. Here, we take a very different virw of eontinyal leaeninn and find insoirstlon in the bramns ability for dimensionality reduxtion [@Pajg2016] to extract meeningful inflvmatioh from its environment and frive behavior. As xuch, we enable such
one task to another. This ability to and information across is referred to @Hassabis2017; The major challenge creating ML models continual learning ability is that they prone to *catastrophic forgetting* [@McClelland1995; @McCloskey1989; @Goodfellow2013; @French1999]. ML models tend to forget knowledge learned from previous tasks when re-trained on new observations corresponding to a (but task. when deep neural network (DNN) is fed with a sequence of tasks, the ability to solve the task will decline significantly after training on the tasks. The typical structure DNNs by design does not the of preserving learned without between tasks or forgetting. In order to overcome catastrophic forgetting, a learning system is required to continuously acquire knowledge from newly fed well as prevent training the new data destroying the existing knowledge. In this a novel approach to continual learning with DNNs addresses the forgetting issue, namely a technique called leverage score sampling (OLSS)*. In OLSS, we progressively the input information learned thus far, along with the input from current task and form condensed data samples. The technique is based the leverage measure, it uses concept of frequent directions in order to connect the series of steps for a sequence of tasks. When thinking about continual major of inspiration is ability of biological brains learn destructive interference between older generalize across this the approach is enabling some of episodic-memory in the network consolidation [@McClelland1995] via replay this is an expensive process and does not well for learning large number of tasks. an alternative, taking inspiration from the neuro-computational models of complex synapses [@BennaFusi2016], work has assigning some form of importance to parameters in DNN and perform task-specific consolidation [@Kirkpatrick2017; @Zenke2017]. Here, we take a very different of learning and inspiration in the ability for dimensionality [@Pang2016] to extract from its and behavior. enable such
 one task to another. This abiliTy to remembEr, leaRn aNd tRaNsfeR infOrmation across TAsks Is referred to as continuaL learNiNG [@ThrUN1995; @RUvolo2013; @hassabiS2017; @paRISi2019]. THe MaJor ChALlEnge fOr cReating mL models wiTh cOnTinual learniNG aBility is thAt tHey are prone tO *caTastroPhIc fORgettIng* [@mcCleLland1995; @MCcloskeY1989; @GoodfellOw2013; @fRench1999]. Ml Models tEND tO forGet the knowledge leARnED from previous tAsks whEn RE-tRAIneD on New observaTiOns coRRespondINg TO A DifFErent (but relatEd) task. SpeciFIcaLly wheN a DeeP Neural NetwoRk (dnN) iS fed with a seQuenCe of tasks, The abiLIty to soLVe the fiRst tasK wiLl dEcliNE sIgNifIcANtlY AfTer TRaiNing on thE fOlLowinG tasKS. tHE typIcaL strUcturE of DNNs by desiGn dOes nOT poSsess The caPabiLiTy of pReservIng prEvIously learned knOwleDge withouT inTeRfeReNce beTWeen taSks Or cAtastroPhic forGEttInG. iN OrDer to overcome catasTrOPHiC forgettIng, a leARnInG System is ReQuiRed tO COntinUousLY aCquire knOwledgE FrOm The newlY fEd data As WelL as To preVEnt tHe traiNing of thE new dATa samples from dEStroying the exIStING kNOwleDge.

in this paper, We prOPose A novEL aPprOAch to ContiNuAL lEArning with DNNs that aDdResses The caTastrophic forGetting issUE, NAmely a teChniQUe CAlled *online levErage Score samplINg (OLSS)*. In oLSS, wE progresSively comPREss the inPut InfOrmAtiON LeArned thus far, aLONg wiTh The inpuT frOm curreNt tAsk And ForM mOre efficiEntly conDeNsEd DaTa sAmpleS. the comprEsSioN tEchNique IS based On the StatIsTiCAl lEverage SCoRES meaSuRe, And iT usEs The coNcepT Of fRequent DirectionS in ORder To CoNnect thE series of compReSsion steps FoR a sEquencE OF tasks.

WhEn thinking about continuaL LearninG, a mAjor sOurcE of inspirAtiOn is thE abILity of BiologIcal bRaIns TO Learn WIThOut DeStructive iNTErfErencE bEtweEn older Memories and generalIZe kNowledge acrosS muLtipLE TaSks. iN tHIs rEgARd, tHE Typical approach Is enabling SoME fOrm of episoDIc-mEmOry in thE networK and cONsolidaTion [@McCleLland1995] via rEpLay oF OLdeR training dAta. HowevEr, this is aN ExpenSIvE procEss And doeS nOt sCale wEll for LEarNing lArge nuMbEr of taSks. As An AlternatIve, taking inspiration froM the neUro-coMpuTational mOdeLS of Complex syNapsEs [@BennaFusI2016], reCenT work Has FOcuseD on aSSiGniNG some Form OF importanCE tO paRAMeTers in a DNN aND PErfOrm taSk-sPEcific SynaPtic consolidation [@kIrkpatrick2017; @ZenkE2017]. HerE, WE taKe a VEry dIfFerent view of coNtiNuAL Learning AnD find inspirAtion in tHe BRains AbilitY for diMensionALItY ReductIon [@PAng2016] To extract MeaNiNGful infOrMaTIon froM its EnVironmEnt and DRive BEHavior. As such, we enAble sUCH
 one task to another. This  ability t o rem emb er,  l earn  and  transfer info r mati on across tasks is ref erred  t o  asc on tinua l learn i ng [ @Th ru n1 995 ;@ Ru volo2 013 ; @Hass abis2017;@Pa ri si2019]. The ma jor challe nge  for creatin g M L mode ls  wi t h con tin ual l earnin g  abili ty is tha tt hey ar e  pronet o  * cata strophic forgetti n g* [@McClelland19 95; @M cC l os k e y19 89;  @Goodfell ow 2013; @French 1 99 9 ] .  ML models tend t o forget th e  kn owledg elea r ned fr om pr ev i ous  tasks when  re- trained o n newo bservat i ons cor respon din g t o ad if fe ren t( but re lat e d)task. Sp ec if icall y wh e n a  dee p n eura l net work (DNN) is  fe d wi t h a  sequ enceof t as ks, t he abi lityto  solve the firs t ta sk will d ecl in e s ig nific a ntly a fte r t raining  on the fol lo w i n gtasks. The typical  s t r uc ture ofDNNs b y  d es i gn doesno t p osse s s  thecapa b il ity of p reserv i ng  p revious ly  learn ed  kn owl edgew itho ut int erferenc e bet w een tasks or c a tastrophic fo r ge t t in g . In  or der to over come cata stro p hi c f o rgett ing,al ea r ning system is requ ir ed toconti nuously acqui re knowled g e from the  new l yf ed data as wel l asto prevent the trai ningof the n ew data s a m ples fro m d est roy ing t he  existing kno w l edge .
In thi s p aper, w e p rop ose  ano vel appro ach to c on ti nu al  le arnin g  with DN Ns  th at  ad dress e s thecatas trop hi cf org ettingi ss u e , na me ly  a t ech ni que c alle d  *o nline l everage s cor e  sam pl in g (OLSS )*. In OLSS,we  progressi ve lycompre s s  the inp ut information learnedt hus far , a longwith  the inpu t f rom cu rre n t task  and f orm m or e e f f icien t l ycon de nsed datas a mpl es. T he  com pressio n technique is bas e d o n the statist ica l le v e ra ges co r esme a sur e ,  and it uses th e conceptof fr equent dir e cti on s in or der toconne c t the s eries ofcompressi on  ste p s  fo r a sequen ce of ta sks.

Whe n  thin k in g abo utcontin ua l l earni ng, am ajo r sou rce of  i nspira tionis  the abi lity of biological brai ns tolearn  wi thout des tru c tiv e interfe renc e betweenold ermemor ies and g ener a li zek nowle dgea cross mul t ip let a sk s. In thisr e g ard , the  ty p ical a ppro ach is enabling s o me form of epi sodi c - mem ory in t he  network and c ons ol i d ation [@ Mc Clelland199 5] via r ep l ay of  older  train ing dat a .  H o wever,  thi s i s an expe nsi ve process  a nd does n ot s ca le wel l forl earn i n g large number o f tas k s . Asa n a ltern at ive, ta k inginspiratio n from theneuro- comp utati onal mo de ls ofcom pl ex synapse s  [@BennaF usi20 16], re ce nt w ork  has f ocus e d  on a ssig ni ngsome form o fi mp or t anc e to  para me ters  in a DNN and perf orm task-sp ec ifi c synapt i cc o nsolidatio n [ @Kirk p a trick2017; @Zen k e2 0 17].Here,we tak e a ver y  di ff erent v iew o f continu al learni n g a nd  fin d inspir at ionin  th ebrains abilityfor dime nsionali t y  reduc tio n  [@P a ng2016 ]to extract m eanin gf u l infor m a tio n fromi tsenv iro nmen t a nd d r ive behavior. As such, w e en ables uch 
 one_task to_another. This ability to_remember, learn_and_transfer information_across_tasks is referred_to as continual_learning [@Thrun1995; @Ruvolo2013; @Hassabis2017;_@Parisi2019]. The major_challenge_for creating ML models with continual learning ability is that they are prone to_*catastrophic_forgetting* [@McClelland1995;_@McCloskey1989;_@Goodfellow2013;_@French1999]. ML models tend to_forget the knowledge learned from_previous tasks_when re-trained on new observations corresponding to a_different_(but related) task._Specifically when a deep neural network (DNN) is fed_with a sequence of tasks, the_ability to solve_the_first_task will decline significantly_after training on the following tasks._The typical structure of DNNs by_design does not possess the capability of_preserving previously learned knowledge without interference_between tasks or catastrophic forgetting._In order_to overcome catastrophic forgetting, a_learning system is_required to_continuously acquire knowledge_from the newly fed data as_well as to_prevent the training of the new_data_samples from destroying_the_existing_knowledge.

In this_paper, we propose_a_novel approach_to_continual learning with DNNs that addresses_the_catastrophic forgetting issue, namely a technique called_*online leverage score sampling_(OLSS)*._In OLSS, we progressively_compress the input information learned_thus far, along with the input_from current_task and_form more efficiently condensed data samples. The compression technique is based_on the statistical leverage scores measure,_and it uses the_concept of_frequent_directions in order_to_connect the_series of compression steps for a sequence_of tasks.

When_thinking about continual learning, a major_source of inspiration is_the_ability of biological brains to learn_without destructive interference between older memories_and generalize knowledge across multiple_tasks._In_this regard, the typical approach_is enabling some form of episodic-memory_in the network_and consolidation [@McClelland1995] via replay of older_training_data. However, this is an expensive_process_and does not scale well for_learning_large_number of tasks. As an_alternative, taking inspiration from the neuro-computational_models of complex synapses [@BennaFusi2016], recent work has focused_on assigning some_form of importance to parameters_in_a_DNN and perform task-specific synaptic consolidation [@Kirkpatrick2017; @Zenke2017]. Here, we_take a_very different view_of continual learning and find inspiration in the brains ability_for dimensionality reduction [@Pang2016] to extract meaningful_information from its environment and drive behavior. As such, we enable_such