@Html.Raw("window.open('") @Url.Action("CallList", "Reports", new { target="_blank" }) @Html.Raw("')");
}

It will print out an open window command and the url has a target of blank to open in a new window.  I put the needed list of id's into the TempData to pass it to the report page.  
So the report gets the needed parameters from the TempData when it is called when the search page reloads.  I also had to make a small modification to the search method to recheck the check boxes in the search results grid so it appears to the end user that only the new page was opened with the requested report.
                IsSelected = searchModel.SelectedArea.Contains(item.ID),

 Q:

Read from Kafka and write to hdfs in parquet

I am new to the BigData eco system and kind of getting started.
I have read several articles about reading a kafka topic using spark streaming but would like to know if it is possible to read from kafka using a spark job instead of streaming?
If yes, could you guys help me in pointing out to some articles or code snippets that can get me started.
My second part of the question is writing to hdfs in parquet format.
Once i read from Kafka, i assume i will have an rdd.
Convert this rdd into a dataframe and then write the dataframe as a parquet file.
Is this the right approach.
Any help appreciated. 
Thanks

A:

You already have a couple of good answers on the topic. 
Just wanted to stress out - be careful to stream directly into a parquet table.
Parquet's performance shines when parquet row group sizes are large enough (for simplicity, you can say file size should be in order of 64-256Mb for example), to take advantage of dictionary compression, bloom filters etc. (one parquet file can have multiple row chunks in it, and normally does have multiple row chunks in each file; although row chunks can't span multiple parquet files)
If you're streaming directly to a parquet table, then you'll end up very likely with a bunch of tiny parquet files (depending on mini-batch size of Spark Streaming, and volume of data). Querying such files can be very slow. Par
@Html. Raw("window.open (' ") @Url. Action("CallList ", " Reports ", new { target="_blank " }) @Html. Raw ("') "); 
 } 

 It will print out an open windowpane instruction and the url own a prey of blank to afford in a new window.   I arrange the want list of i d's into the TempData to pass it to the report page.  
 So the reputation gets the needed argument from the TempData when it is called when the search page reloads.   I also had to make a minor modification to the search method to recheck the check corner in the search results grid so it appears to the end user that merely the new page was opened with the requested report. 
                 IsSelected = searchModel. SelectedArea. Contains(item. ID), 

  Q: 

 Read from Kafka and spell to hdfs in parquet 

 I am new to the BigData eco system and kind of getting started. 
 I have read several articles about reading a kafka topic using spark streaming but would like to know if it is possible to read from kafka using a spark occupation instead of streaming? 
 If yes, could you guy help me in pointing out to some articles or code snippets that can have me started. 
 My second part of the question is writing to hdfs in parquet format. 
 Once i understand from Kafka, i assume i will have an rdd. 
 Convert this rdd into a dataframe and then write the dataframe as a parquet file. 
 Is this the right approach. 
 Any help prize. 
 Thanks 

 A: 

 You already have a couple of good answers on the topic. 
 Just wanted to try out - be careful to pour directly into a parquet table. 
 Parquet's performance shines when parquet rowing group sizes are large enough (for simplicity, you can suppose file size should be in order of 64 - 256Mb for example), to take advantage of dictionary compression, efflorescence filters etc. (one parquet file can take multiple row ball in it, and normally does have multiple quarrel chunks in each file; although quarrel chunks can't cross multiple parquet files) 
 If you're streaming directly to a parquet board, then you'll end up very likely with a crowd of tiny parquet files (depending on mini - batch size of Spark Streaming, and book of data). Querying such files can be very slow. Par
@Htmp.Raw("window.open('") @Url.Action("GallList", "Reports", new { terget="_blznk" }) @Htmu.Raw("')");
}

It will print out an opei wibdow xommand and the url har a targen of blanj to ipen in a isw window.  I lmt thz ieeded list of ld's into tha TempData to [ars it to the report page.  
So the repowt gets tje needed paraietegs froj the TempData when it is called wgen the search page teloads.  I also had to makf a dmall modification to the seatdh iwthod to recfeck the check boxes ih the search results grid so it appecrs to the gue udgr that only the gew page was opened fith thr requested reiort.
                IsSelectev = searchModel.SelectgdArea.Contdius(item.ID),

 Q:

Read from Kqfja ang wrhte gi hafs ii pzrquet

L aj new to tge BigData wco system and kind os getting starfed.
I hwvq read several articles about reading a karka topic using spark srreaming but would line to knor if it is possible to read from kafka using a spdrk jkc iuwtead if streaming?
If yes, could you guys help me in poyhtonn out to some arbicles or code snilpftx that can get me stcdtsd.
My second part ov the qoestiob is writyng yo hdfs in parquet format.
Onxe i read frjn Kafka, i assume i will have cn rdd.
Vonvett this rdd into a datayrame znd then wrlte the dzgaframe as a parduen fine.
Is this the right approash.
Any hel' appxeciated. 
Thamks

A:

Yot already jave a couple of good andwers ot the topif. 
Just wanted to stress out - be careful to stteak dprectly iuto a iarquet table.
Pawquet's performcnce shiues whdn parquet row grmup sizes awe large enoucj (for simplirity, you san way dile sixd should be in order of 64-256Mb for exqmple), to take advaktage kf dictionary cimptwssion, bloom foltdrs enc. (pnq parquet fila cav hxfe muutiple row ghuvks on it, and normally dmes gave multiple row vhmnks in eqch file; although row chunks can't span lulti'le pacquet gilgs)
If you're streaming directly tk a parquft bable, then yot'll cnd op very likzly with a bunch of tiny parquet files (dxpending on mini-batch suze of Spark Streamnnn, and volume of dwta). Queryhng such files can bw very slow. Par
@Html.Raw("window.open('") @Url.Action("CallList", "Reports", new { target="_blank" }) It print out open window command target blank to open a new window. put the needed list of id's the TempData to pass it to the report page. So the report gets needed parameters from the TempData when it is called when the search page I had make small modification to the search method to recheck the check boxes in the search results grid it appears to the end user that only new page was opened the requested report. IsSelected = Q: from Kafka write hdfs parquet I am to the BigData eco system and kind of getting started. I have read several articles about reading kafka topic streaming but like know it is possible from kafka using a spark job If yes, could you guys help me in out to articles or code snippets that can me started. My second part of the question writing to hdfs in parquet format. Once i read from Kafka, i assume i will rdd. Convert this rdd a dataframe and write dataframe a file. Is the right approach. Any help appreciated. Thanks A: You already have couple of good answers on the topic. Just wanted to - careful to stream into a parquet table. performance when parquet row group large (for say size be in order of for example), to take advantage dictionary compression, bloom filters have multiple row chunks in it, and normally have multiple row chunks in each file; row chunks can't span multiple parquet files) If you're streaming directly to parquet table, end up very likely with a bunch of parquet files (depending on size of Spark Streaming, and volume of data). Querying files be very Par
@Html.Raw("window.open('") @Url.ActioN("CallList", "REportS", neW { taRgEt="_blAnk" }) @HTml.Raw("')");
}

It will prINt ouT an open window command anD the uRl HAs a tARgEt of bLank to oPEn IN A neW wInDow.  i pUT tHe neeDed List of iD's into the TEmpdaTa to pass it to THe Report page.  
so tHe report gets The Needed PaRamETers fRom The TeMpData WHen it iS called whEn THe searCH page reLOAdS.  I alSo had to make a small MOdIFication to the sEarch mEtHOd TO RecHecK the check bOxEs in tHE search REsULTS grID so it appears tO the end user THat Only thE nEw pAGe was oPened WiTH thE requested rEporT.
                IsSelectEd = searCHModel.SELectedARea.ConTaiNs(iTem.Id),

 q:

REaD frOm kAfkA AnD wrITe tO hdfs in pArQuEt

I am New tO THE bigDAta Eco sYstem And kind of gettIng StarTEd.
I Have rEad seVeraL aRticlEs abouT readInG a kafka topic usiNg spArk streamIng BuT woUlD like TO know iF it Is pOssible To read fROm kAfKA USiNg a spark job instead Of STReAming?
If yEs, coulD YoU gUYs help me In PoiNtinG OUt to sOme aRTiCles or coDe snipPEtS tHat can gEt Me starTeD.
My SecOnd paRT of tHe quesTion is wrIting TO hdfs in parquet FOrmat.
Once i reaD FrOM kaFKa, i aSsuMe i will have An rdD.
convErt tHIs Rdd INto a dAtafrAmE AnD Then write the datafraMe As a parQuet fIle.
Is this the rIght approaCH.
aNy help apPrecIAtED. 
Thanks

A:

You alrEady hAve a couple OF good ansWers oN the topiC. 
Just wantED To stress Out - Be cAreFul TO StReam directly iNTO a paRqUet tablE.
PaRquet's pErfOrmAncE shInEs when parQuet row gRoUp SiZeS arE largE Enough (foR sImpLiCitY, you cAN say fiLe sizE shoUlD bE In oRder of 64-256MB FoR EXampLe), To Take AdvAnTage oF dicTIonAry compRession, blOom FIlteRs EtC. (one parQuet file can haVe Multiple roW cHunKs in it, AND normallY does have multiple row chuNKs in eacH fiLe; altHougH row chunkS caN't span MulTIple paRquet fIles)
IF yOu'rE STreamING dIreCtLy to a parquET TabLe, theN yOu'll End up veRy likely with a bunch OF tiNy parquet fileS (dePendING oN miNI-bATch SiZE of sPArk Streaming, and Volume of daTa). qUeRying such fILes CaN be very Slow. Par
@Html.Raw("window.open('")  @Url.Acti on("C all Lis t" , "R epor ts", new { tar g et=" _blank" }) @Html.Raw(" ')"); 
} 
Itw il l pri nt outa no p enwi nd owco m ma nd an d t he urlhas a targ etof  blank to op e nin a new w ind ow.  I put t heneeded  l ist of id 'sintothe Te m pDatato pass i tt o ther eport p a g e.   
S o the report gets th e  needed parame ters f ro m  t h e  Te mpD ata when i tis ca l led whe n  t h e sea r ch page reloa ds.  I also had  to ma ke  as mall m odifi ca t ion  to the sea rchmethod to  reche c k the c h eck box es inthe  se arch re su lts  g r ids oita ppe ars to t he  e nd us er t h a t only  th e ne w pag e was openedwit h th e  re quest ed re port .             Is Selected = sear chMo del.Selec ted Ar ea. Co ntain s (item. ID) ,
 Q:

Re ad from Kaf ka a n dwrite to hdfs in p ar q u et 

I am n ew tot he  B i gData ec osys tema n d kin d of ge tting st arted. Iha ve read  s everal  a rti cle s abo u t re adinga kafkatopic using spark st r eaming but wo u ld l ik e  tokno w if it isposs i bleto r e ad  fr o m kaf ka us in g  a spark job instead o fstream ing?If yes, could  you guysh e l p me inpoin t in g  out to some a rticl es or code snippets  that  can get  me start e d .
My sec ond  pa rtoft h equestion is w r i ting  t o hdfsinparquet  fo rma t.Onc ei read fr om Kafka ,ias su mei wil l  have an  r dd. 
C onv ert t h is rdd  into  a d at af r ame  and th e nw r iteth edata fra me  as a  par q uet  file.Is this t her ight  a pp roach.Any help appr ec iated. 
Th an ks
A:

Y o u  already  have a couple of gooda nswersonthe t opic . 
Just w ant ed tostr e ss out  - becaref ul  to s tream d ir ect ly  into a pa r q uet  tabl e. 
Par quet'sperformance shines whe n parquet row  gr oups i ze s a r el arg ee nou g h  (for simplicit y, you can  s a yfile sizes hou ld  be inorder o f 64- 2 56Mb fo r example ), to tak eadva n t age  of dictio nary com pression, bloom fi lters  et c. (on epar quetfile c a n h ave m ultipl erow ch unksin  it, and  normally does have mul tiplerow c hun ks in eac h f i le;  although  row  chunks ca n't  sp an mu lti p le pa rque t  f ile s )
Ifyou' r e streami n gdir e c tl y to a parq u e t  ta ble,the n  you'l l en d up very likelyw ith a bunch of  tin y par que t  fil es  (depending on  mi ni - b atch siz eof Spark St reaming,  a n d vol ume of  data) . Query i n gs uch fi lescan  be veryslo w. Par
@Html.Raw("window.open('") @Url.Action("CallList",_"Reports", new_{ target="_blank" }) @Html.Raw("')");
}

It_will print_out_an open_window_command and the_url has a_target of blank to_open in a_new_window.  I put the needed list of id's into the TempData to pass_it_to the_report_page._ 
So the report gets_the needed parameters from the_TempData when_it is called when the search page reloads.__I also had_to make a small modification to the search method_to recheck the check boxes in_the search results_grid_so_it appears to the_end user that only the new_page was opened with the requested_report.
      _     _   IsSelected =_searchModel.SelectedArea.Contains(item.ID),

 Q:

Read_from Kafka and write to_hdfs in parquet

I_am new_to the BigData_eco system and kind of getting_started.
I have read_several articles about reading a kafka_topic_using spark streaming_but_would_like to_know if it_is_possible to_read_from kafka using a spark job_instead_of streaming?
If yes, could you guys help_me in pointing out_to_some articles or code_snippets that can get me_started.
My second part of the question_is writing_to hdfs_in parquet format.
Once i read from Kafka, i assume i will_have an rdd.
Convert this rdd into_a dataframe and then_write the_dataframe_as a parquet_file.
Is_this the_right approach.
Any help appreciated. 
Thanks

A:

You already have_a couple_of good answers on the topic._
Just wanted to stress_out_- be careful to stream directly_into a parquet table.
Parquet's performance shines_when parquet row group sizes_are_large_enough (for simplicity, you can_say file size should be in_order of 64-256Mb_for example), to take advantage of dictionary_compression,_bloom filters etc. (one parquet file_can_have multiple row chunks in it,_and_normally_does have multiple row chunks_in each file; although row chunks_can't span multiple parquet files)
If you're streaming directly to_a parquet table,_then you'll end up very_likely_with_a bunch of tiny parquet files (depending on mini-batch size_of Spark_Streaming, and volume_of data). Querying such files can be very slow. Par