 => self! body case Failure ( err ) => self! Status. Failure ( err ) } def getAllLinks ( content : String ) : Iterator [ String ] = { Jsoup. parse ( content, this. url ). select ( "a[href]" ). iterator (). asScala. map ( _. absUrl ( "href" )) } def receive = { case body : String => getAllLinks ( body ). filter ( link => link!= null && link. length > 0 ). filter ( link =>! link. contains ( "mailto" )). filter ( link => currentHost == new URL ( link ). getHost ). foreach ( context. parent! LinkChecker. CheckUrl ( _, depth )) stop case _: Status.Failure => stop () } def stop () : Unit = { context. parent! Done context. stop ( self ) } }

As soon as the actor is initialised the actor will initiate a call to retrieve the contents of the URL - WebClient.get(url). If the call completes successfully the HTML body is sent to the actor otherwise a Failure message is sent. Even though we could have called directly the receive partial function we always communicate with the actor using messages - messages are kings! Calling the receive partial function directly would break the guarantees provided by the Actor System. Always, always, always (did I say always?) communicate with Actors through messages.

When the actor receives the content body, all links are retrieved by calling the getAllLinks function and an individual CheckUrl message is sent to the LinkChecker actor - context.parent! LinkChecker.CheckUrl(_, depth). Once all links are communicated back to the LinkChecker, the Getter will send a Done message to the LinkChecker and die ( stop function). Note that the getAllLinks filters out invalid links, mailto links, and also links which are not within the root domain.

If the Getter receives a Failure message the Getter will send a Done message and die.

Link Checker

Web pages are filled with links and these links can create cycles. In order to prevent our crawler from crawling the web ad infinitum we will track all links and make sure that they are unique. The LinkChecker is implemented as follows:

object LinkChecker { case class CheckUrl ( url : String, depth : Int ){} case class Result ( url : String, links :
= > self! body case Failure (err) = > self! Status. Failure (err) } def getAllLinks (contentedness: chain): Iterator [ String ] = { Jsoup. parse (content, this. url). select (" a[href ] "). iterator (). asScala. map (_. absUrl (" href ") ) } def receive = { lawsuit body: String = > getAllLinks (torso). filter (link = > link!= null & & link. length > 0). filter (associate = >! link. contains (" mailto ") ). filter (yoke = > currentHost = = new URL (link). getHost). foreach (context. parent! LinkChecker. CheckUrl (_, astuteness) ) stop case _: Status. Failure = > stop () } def stop (): Unit = { context. parent! Done context. discontinue (self) } } 

 As soon as the actor is initialised the actor will initiate a call option to retrieve the contents of the URL - WebClient.get(url). If the call completes successfully the HTML torso is sent to the actor otherwise a Failure message is sent. Even though we could have called immediately the receive partial function we always communicate with the actor using messages - message are kings! Calling the receive partial function directly would break the guarantees provided by the Actor System. Always, always, constantly (did I say always ?) commune with actor through messages. 

 When the actor receives the content soundbox, all links are retrieved by calling the getAllLinks function and an individual CheckUrl message is commit to the LinkChecker actor - context.parent! LinkChecker. CheckUrl (_, depth). Once all links are communicated back to the LinkChecker, the Getter will send a Done message to the LinkChecker and die (barricade function). Note that the getAllLinks filters out invalid links, mailto connection, and besides links which are not within the root domain. 

 If the Getter receives a Failure message the Getter will send a Done message and die. 

 Link Checker 

 world wide web pages are filled with links and these links can produce cycles. In decree to prevent our crawler from crawl the web ad infinitum we will chase all links and reach sure that they are singular. The LinkChecker is implemented as follows: 

 object LinkChecker { case class CheckUrl (url: String, depth: Int) { } casing class Result (url: String, links:
 => sflf! body case Failure ( evr ) => self! Status. Failurx ( err ) } def gegAllLinks ( content : String ) : Ireratir [ String ] = { Jsoup. pafse ( contvnt, this. yrl ). welect ( "a[hcsf]" ). itevctor (). wsSccle. map ( _. absUrl ( "href" )) } dex receive = { cave bldy : String => getAllLinks ( body ). fileer ( limk => link!= null && lynk. kqngtg > 0 ). filter ( link =>! link. contains ( "jailto" )). filter ( link => currentHost == new URL ( lijk ). hetHost ). foreach ( fontext. parght! JunkChecker. CfeckUrl ( _, depth )) stop dase _: Status.Failure => stop () } def stop () : Unit = { xobtedj. parent! Donx contvxt. stop ( seln ) } }

As voon as the actor is lnitieliswd the actor will iniviate a call to retryeve the woutents of the URL - WevCoient.cet(usl). Iw thd czlk domplehes successfumly the HTMO body is sent to tne qctor otherwiae a Fwijure message is sent. Even though we cound gave called directly thw receive partial funftion we wlways communicate with the actor using messages - messegds cvc kivts! Calling the receive partial function directlr wpukd break the gmarantees provided bj yre Actor Systgm. Alwava, zlways, always (did L say ajways?) communicwte eith Actors through messagew.

When the acnor eeceives the conteut body, all uinkx are retrieved by calling tke getZllLinks fujction ans an individual CfeclUsl message is sent to the JinkCheckxr accor - congext.karent! JinkCheckeg.CheckUrl(_, depth). Once all linkd dre communlcated back to the LinkChecker, vie Getter wilk veng a Done messsge to the LigkChecker and bie ( sto' funcgion). Note nhat the jetAllLinks silters out itgalid links, kailto lynks, and also lkvks which are mot withiu the root domain.

If the Gejtsr receives a Fcnloee message the Gegtew aikl vend a Done kessxge snd dke.

Link Checjdr

Wen pages are filled whth minks and these limkf can crgate cyclqs. In order tp prevent our crawper fcom crewling thg web ad infinitum we will traci all linns wnd make surq thqt they are bnique. The LinkChecker is implemented as follows:

object LinjChecker { case clasx CheckUrl ( nrl : Sering, depdh : Int ){} case class Eesult ( url : Strikg, links :
=> self! body case Failure ( err self! Failure ( ) } def ) Iterator [ String = { Jsoup. ( content, this. url ). select "a[href]" ). iterator (). asScala. map ( _. absUrl ( "href" )) } receive = { case body : String => getAllLinks ( body ). filter link link!= && length > 0 ). filter ( link =>! link. contains ( "mailto" )). filter ( link currentHost == new URL ( link ). getHost foreach ( context. parent! CheckUrl ( _, depth )) case Status.Failure => () def () : Unit { context. parent! Done context. stop ( self ) } } As soon as the actor is the actor a call retrieve contents the URL - the call completes successfully the HTML to the actor otherwise a Failure message is Even though could have called directly the receive function we always communicate with the actor using - messages are kings! Calling the receive partial function directly would break the guarantees provided Actor System. Always, always, (did I say communicate Actors messages. the actor the content body, all links are retrieved by calling the getAllLinks and an individual CheckUrl message is sent to the LinkChecker context.parent! depth). Once all are communicated back to LinkChecker, Getter will send a to LinkChecker stop Note the getAllLinks filters out links, mailto links, and also which are not within Getter receives a Failure message the Getter will a Done message and die. Link Checker pages are filled with links and these links can create cycles. In to prevent from crawling the web ad infinitum we will all links and make that they are unique. The LinkChecker is implemented as object { case CheckUrl ( url String, depth : ){} case class url : links
 => self! body case Failure ( err ) => selF! Status. FaiLure ( eRr ) } dEf gEtallLInks ( Content : String ) : ITEratOr [ String ] = { Jsoup. parse ( contEnt, thIs. URl ). seLEcT ( "a[hreF]" ). iteratOR (). aSsCalA. mAp ( _. AbsurL ( "HrEf" )) } def RecEive = { casE body : StrinG => geTALlLinks ( body ). fILtEr ( link => link!= NulL && link. length > 0 ). fIltEr ( link =>! LiNk. cONtainS ( "maIlto" )). fIlter ( lINk => currEntHost == neW Url ( link ). gETHost ). foREAcH ( conText. parent! LinkCheCKeR. checkUrl ( _, depth )) sTop casE _: STAtUS.faiLurE => stop () } def stOp () : unit = { cONtext. paREnT! dONe cONtext. stop ( self ) } }

as soon as the ACtoR is iniTiAliSEd the aCtor wIlL IniTiate a call tO retRieve the cOntentS Of the URl - webClieNt.get(uRl). IF thE calL CoMpLetEs SUccESsFulLY thE HTML bodY iS sEnt to The aCTOR OtheRwiSe a FAilurE message is senT. EvEn thOUgh We couLd havE calLeD direCtly thE receIvE partial functioN we aLways commUniCaTe wItH the aCTor usiNg mEssAges - mesSages arE KinGs! cALLiNg the receive partiaL fUNCtIon direcTly wouLD bReAK the guarAnTeeS proVIDed by The ACToR System. ALways, aLWaYs, Always (dId i say alWaYs?) cOmmUnicaTE witH ActorS through MessaGEs.

When the actor REceives the conTEnT BOdY, All lInkS are retrievEd by CAlliNg thE GeTAlLlinks FunctIoN AnD An individual CheckUrL mEssage Is senT to the LinkCheCker actor - cONTExt.parenT! LinKchECker.CheckUrl(_, dePth). OnCe all links ARe communIcateD back to tHe LinkCheCKEr, the GetTer WilL seNd a dONe Message to the LINKCheCkEr and diE ( stOp functIon). notE thAt tHe GetAllLinKs filterS oUt InVaLid Links, MAilto linKs, And AlSo lInks wHIch are Not wiThin ThE rOOt dOmain.

If THe gETter ReCeIves A FaIlUre meSsagE The getter wIll send a DOne MEssaGe AnD die.

LinK Checker

Web paGeS are filled WiTh lInks anD THese linkS can create cycles. In order TO prevenT ouR crawLer fRom crawliNg tHe web aD inFInitum We will Track AlL liNKS and mAKE sUre ThAt they are uNIQue. the LiNkchecKer is imPlemented as follows:

OBjeCt LinkChecker { CasE claSS chEckuRl ( URl : STrINg, dEPTh : Int ){} case class REsult ( url : StRiNG, lInks :
 => self! body case Failur e ( err )=> se lf!  St at us.Fail ure ( err ) }d ef g etAllLinks ( content :  Stri ng ) :I te rator  [ Stri n g] = {  J so up.  p a rs e ( c ont ent, th is. url ).  se le ct ( "a[href ] "). iterato r ( ). asScala.map  ( _.ab sUr l  ( "h ref " ))} defr eceive  = { case  b o dy : S t ring => g et AllL inks ( body ). fi l te r  ( link => lin k!= nu ll && l ink . l ength > 0).  filt e r ( lin k  = > ! lin k . contains ("mailto" )) .  fi lter (  l ink => cur rentH os t  ==  new URL (link  ). getHo st ).f oreach(  contex t. par ent ! L inkC h ec ke r.Ch e ckU r l( _ ,  de pth )) s to pcase_: S t a t u s.Fa ilu re = > sto p () } def st op() : Uni t = {  cont ext.  p arent ! Done  cont ex t. stop ( self) }}

As soo n a sthe  a ctori s init ial ise d the a ctor wi l l i ni t i a te  a call to retriev et h econtents  of th e  U RL - WebCli en t.g et(u r l ). If  the ca ll compl etes s u cc es sfullyth e HTML  b ody  is  sent to t he act or other wisea  Failure messa g e is sent. Ev e nt h ou g h we  co uld have ca lled dire ctly th e r e ceive  part ia l  f u nction we always co mm unicat e wit h the actor u sing messa g e s  - messa gesa re kings! Calling  thereceive pa r tial fun ction  directl y would b r e ak the g uar ant ees  pr o v id ed by the Act o r  Sys te m. Alwa ys,  always , a lwa ys(di dI say alw ays?) co mm un ic at e w ith A c tors thr ou ghme ssa ges.When t he ac torre ce i ves  the co n te n t  bod y,  a ll l ink sare r etri e ved  by cal ling theget A llLi nk sfunctio n and an indi vi dual Check Ur l m essage i s sent t o the LinkChecker actor - conte xt. paren t! L inkChecke r.C heckUr l(_ ,  depth ). Onc e all  l ink s are c o m mu nic at ed back to t heLinkC he cker , the G etter will send aD one  message to t heLink C h ec ker an d  di e(  st o p  function). Not e that the  g e tA llLinks fi l ter sout inv alid li nks,m ailto l inks, and  also lin ks  whi c h  ar e not with in the r oot domai n .

If th e Get ter  recei ve s a  Fail ure me s sag e the  Gette rwill s end a  D one mess age and die.

Link Chec ker

W eb pa ges  are fill edw ith  links an d th ese linkscan  cr eatecyc l es. I n or d er  to preve nt o u r crawler fr omc r aw ling the we b a d i nfini tum we wil l tr ack all links and make sure that  the y are  un i que.  T he LinkChecker  is  i m p lemented  a s follows:
objectLi n kChec ker {case c lass Ch e c kU r l ( ur l :Str ing, dept h :  I n t ){} c as ec lass R esul t( url: Stri n g, l i n ks :
 =>_self! body_case Failure ( err_) =>_self!_Status. Failure_(_err ) }_def getAllLinks (_content : String )_: Iterator [_String_] = { Jsoup. parse ( content, this. url ). select ( "a[href]" )._iterator_(). asScala._map_(__. absUrl ( "href" ))_} def receive = {_case body_: String => getAllLinks ( body ). filter_(_link => link!=_null && link. length > 0 ). filter (_link =>! link. contains ( "mailto"_)). filter (_link_=>_currentHost == new URL_( link ). getHost ). foreach_( context. parent! LinkChecker. CheckUrl (__, depth )) stop case _: Status.Failure_=> stop () } def stop_() : Unit = {_context. parent!_Done context. stop ( self_) } }

As_soon as_the actor is_initialised the actor will initiate a_call to retrieve_the contents of the URL -_WebClient.get(url)._If the call_completes_successfully_the HTML_body is sent_to_the actor_otherwise_a Failure message is sent. Even_though_we could have called directly the receive_partial function we always_communicate_with the actor using_messages - messages are kings!_Calling the receive partial function directly_would break_the guarantees_provided by the Actor System. Always, always, always (did I say_always?) communicate with Actors through messages.

When_the actor receives the_content body,_all_links are retrieved_by_calling the_getAllLinks function and an individual CheckUrl message_is sent_to the LinkChecker actor - context.parent!_LinkChecker.CheckUrl(_, depth). Once all_links_are communicated back to the LinkChecker,_the Getter will send a Done_message to the LinkChecker and_die_(_stop function). Note that the_getAllLinks filters out invalid links, mailto_links, and also_links which are not within the root_domain.

If_the Getter receives a Failure message_the_Getter will send a Done message_and_die.

Link_Checker

Web pages are filled with_links and these links can create_cycles. In order to prevent our crawler from crawling_the web ad_infinitum we will track all_links_and_make sure that they are unique. The LinkChecker is implemented_as follows:

object_LinkChecker { case_class CheckUrl ( url : String, depth : Int ){}_case class Result ( url : String,_links :