 at work more quickly.

EPW Chairman John Barrasso (R-Wyo.) has said he planned to await the results of a White House review of Pruitt’s conduct and would not comment on multiple occasions this week on when the administrator would return to his committee.

“He was just here earlier this year and answered questions for 2½ hours, but I expect him to come back again,” Barrasso told reporters.

 As you may have seen in previous blog posts, we have a Warboard in our office which shows the current status of the servers we manage. Most of our customers are using our new alerting stack, but some have their own monitoring solutions which we want to integrate with. One of these was Prometheus. This blog post covers how we transformed raw Prometheus values into percentages which we could display on our warboard and create alerts against.

Integrating with Prometheus

In order to get a summary of the data from Prometheus to display on the Warboard we first needed to look at what information the Node Exporter provided and how it was tagged. Node Exporter is the software which makes the raw server stats available for Prometheus to collect. Given that our primary concerns are CPU, memory, disk IO and disk space usage we needed to construct queries to calculate them as percentages to be displayed on the Warboard.

Prometheus makes the data accessible through its API on "/api/v1/query?query=<query>". Most of the syntax is fairly logical with the general rule being that if we take an average or maximum value we need to specify "by (instance)" in order to keep each server separate. Node Exporter mostly returns raw values from the kernel rather than trying to manipulate them. This is nice as it gives you freedom to decide how to use the data but does mean we have to give our queries a little extra consideration:

CPU

(1 - avg(irate(node_cpu{mode="idle"}[10m])) by (instance)) * 100

CPU usage is being reported as an integer that increases over time so we need to calculate the current percentage of usage ourselves. Fortunately Prometheus has the rate and irate functions for us. Since rate is mostly for use in calculating whether or not alert thresholds have been crossed and we are just trying to display the most recent data, irate seems a better fit. We are currently taking data over the last 10 minutes to ensure we get data for all
at work more quickly. 

 EPW Chairman John Barrasso (R - Wyo .) has said he planned to expect the result of a White House review of Pruitt ’s conduct and would not gloss on multiple occasions this workweek on when the administrator would return to his committee. 

 “ He was just here in the first place this class and answered questions for 2½ hours, but I expect him to come back again, ” Barrasso order reporters. 

  As you may have seen in former blog posts, we have a Warboard in our office which shows the current status of the servers we pull off. Most of our customers are using our new alarm stack, but some have their own monitoring solution which we want to desegregate with. One of these was Prometheus. This blog mail covers how we transformed raw Prometheus value into percentages which we could display on our warboard and create alerts against. 

 Integrating with Prometheus 

 In club to induce a summary of the data from Prometheus to display on the Warboard we first needed to look at what information the Node Exporter provided and how it was tagged. Node Exporter is the software which make the raw server stats available for Prometheus to gather. pass that our primary concerns are CPU, memory, disk IO and disk space usage we needed to construct question to calculate them as percentages to be displayed on the Warboard. 

 Prometheus makes the data accessible through its API on " /api / v1 / query?query=<query > ". Most of the syntax is fairly logical with the general rule being that if we take an average or maximum value we necessitate to specify " by (case) " in order to keep each server separate. Node Exporter mostly return raw values from the kernel rather than judge to rig them. This is nice as it gives you exemption to decide how to use the data but does mean we have to give our question a little extra consideration: 

 CPU 

 (1 - avg(irate(node_cpu{mode="idle"}[10 m ]) ) by (case) ) * 100 

 CPU custom is being reported as an integer that increases over time so we need to account the current percentage of custom ourselves. Fortunately Prometheus has the rate and irate functions for us. Since rate is mostly for function in calculating whether or not alert threshold have been crossed and we are just trying to display the most recent datum, irate seems a better fit. We are currently taking data over the last 10 minutes to guarantee we arrive data for all
 at work more quickly.

EPW Chxirman John Bartawso (R-Wbo.) has aaid he olanned to await the results od a Wyite House review of Pfuitt’s cojduct ane wonld not comment on multiije odgasious this week on ehen the agministrator wmuud return to his committee.

“He was just here esrpier this year and wnswsgeb questions for 2½ hours, but I exlect hik to come bacl again,” Barrasso told repogterd.

 As you may have deen in preciouf blog posts, de have a Warboard in kur office which shows the currdnt scatus of thg aegeers we manege. Moft of our customers dre usimg our new alevting stqck, but some have themr own monitoring sojutions wvieh we want to integrare with. One of gyesd wzs Pdomethfus. This blog post coverw how we transformec wqw Prometheus valuef ynto percentages which we could display on our warboard and creatw alerts against.

Integtating witr Prometheus

In order to get a summary of the data from Oronebheur tl display on the Warboard we first needed to jkol st what informction the Node Ecplrygr provided ana how nf sas tagged. Node Exoorter ys thw softwart whivh makes the raw server stars available dor Prometheus to eollect. Giveu that our lrimary concerns are CPB, memody, disk IO wnd disk aoace usage we neddec do construct queries to cajculate tiem ax percevtaggs to bq displayef on bve Warboard.

Promethfus mckes dhe data afcessible through its API on "/apm/t1/query?query=<qugry>". Movt of thz syntsx is fairly jogical with tke genercl ruld being thzt if wx take an avqrage or maxikom value we nxed to spqcift "by (instanzd)" in order to leep each server separate. Node Exportcr morfly returns raw vaoues from the krrndl wanhec thag trying to mdnipjlagr theo. This is nlce as ot gives you freedom to secide how to use yhc data buj does mewn we have to give our queries w litvle exvra comsiqeration:

CPU

(1 - avg(irate(node_cpu{mose="idle"}[10m])) bj (ikstance)) * 100

CPU tsagc is being repprted as an integer that increases over time so we need jo calculate the currgnt percentage of ufage oursalves. Fortunately Primetheus has the vate and irate functiohs for us. Slnce rate is mostly for use in calculating whether or not alert thresholdw have been crosses anc we drz jbst tryyng vo display the mosn recent data, irate seems a bettxr fit. We drz currently taking data over yhd last 10 minutgs to ensure we get data fkr all
at work more quickly. EPW Chairman John has he planned await the results of conduct and would comment on multiple this week on when the administrator return to his committee. “He was just here earlier this year and answered for 2½ hours, but I expect him to come back again,” Barrasso told As may seen previous blog posts, we have a Warboard in our office which shows the current status of servers we manage. Most of our customers are our new alerting stack, some have their own monitoring which want to with. of was Prometheus. This post covers how we transformed raw Prometheus values into percentages which we could display on our warboard create alerts with Prometheus order get summary of the Prometheus to display on the Warboard to look at what information the Node Exporter and how was tagged. Node Exporter is the which makes the raw server stats available for to collect. Given that our primary concerns are CPU, memory, disk IO and disk space needed to construct queries calculate them as to displayed the Prometheus makes data accessible through its API on "/api/v1/query?query=<query>". Most of the syntax fairly logical with the general rule being that if we average maximum value we to specify "by (instance)" order keep each server separate. mostly raw kernel than to manipulate them. This nice as it gives you to decide how to mean we have to give our queries a extra consideration: CPU (1 - avg(irate(node_cpu{mode="idle"}[10m])) by * 100 CPU usage is being reported as an integer that increases time so to calculate the current percentage of usage ourselves. Prometheus has the rate irate functions for us. Since rate is mostly for in whether or alert thresholds have crossed and we just trying to most recent irate a are currently taking data over the 10 to ensure we get data all
 at work more quickly.

EPW ChairMan John BarRasso (r-WyO.) haS sAid hE plaNned to await the REsulTs of a White House review oF PruiTt’S CondUCt And woUld not cOMmENT on MuLtIplE oCCaSions ThiS week on When the admIniStRator would reTUrN to his commIttEe.

“He was just hEre EarlieR tHis YEar anD anSwereD questIOns for 2½ Hours, but I ExPEct him TO come baCK AgAin,” BArrasso told reportERs.

 aS you may have seeN in preViOUs BLOg pOstS, we have a WaRbOard iN Our offiCE wHICH shOWs the current sTatus of the sERveRs we maNaGe. MOSt of ouR custOmERs aRe using our nEw alErting staCk, but sOMe have tHEir own mOnitorIng SolUtioNS wHiCh wE wANt tO InTegRAte With. One oF tHeSe was promETHEUs. ThIs bLog pOst coVers how we tranSfoRmed RAw PRometHeus vAlueS iNto peRcentaGes whIcH we could display On ouR warboard And CrEatE aLerts AGainst.

intEgrAting wiTh PromeTHeuS

IN ORDeR to get a summary of thE dATA fRom PromeTheus tO DiSpLAy on the WArBoaRd we FIRst neEded TO lOok at whaT inforMAtIoN the NodE EXporteR pRovIdeD and hOW it wAs taggEd. Node ExPorteR Is the software wHIch makes the raW SeRVEr STats AvaIlable for PrOmetHEus tO colLEcT. GiVEn thaT our pRiMArY Concerns are CPU, memorY, dIsk IO aNd disK space usage we Needed to coNSTRuct querIes tO CaLCulate them as peRcentAges to be diSPlayed on The WaRboard.

PrOmetheus mAKEs the datA acCesSibLe tHROuGh its API on "/api/V1/QUery?QuEry=<querY>". MoSt of the SynTax Is fAirLy Logical wiTh the genErAl RuLe BeiNg thaT If we take An AveRaGe oR maxiMUm valuE we neEd to SpEcIFy "bY (instanCE)" iN ORder To KeEp eaCh sErVer seParaTE. NoDe ExporTer mostly RetURns rAw VaLues froM the kernel ratHeR than tryinG tO maNipulaTE Them. This Is nice as it gives you freedOM to deciDe hOw to uSe thE data but dOes Mean we HavE To give Our queRies a LiTtlE EXtra cONSiDerAtIon:

CPU

(1 - avg(iRATe(nOde_cpU{mOde="iDle"}[10m])) by (iNstance)) * 100

CPU usage is bEIng Reported as an iNteGer tHAT iNcrEAsES ovEr TIme SO We need to calculaTe the curreNt PErCentage of uSAge OuRselves. fortunaTely PROmetheuS has the raTe and iratE fUnctIONs fOr us. Since rAte is mosTly for use IN calcULaTing wHetHer or nOt AleRt thrEsholdS HavE been CrosseD aNd we arE just TrYing to diSplay the most recent data, iRate seEms a bEttEr fit. We arE cuRRenTly taking Data Over the lasT 10 miNutEs to eNsuRE we geT datA FoR alL
 at work more quickly.

EP W Chairman  John  Ba rra ss o (R -Wyo .) has said he plan ned to await the resul ts of  a Whit e  H ousereviewo fP r uit t’ scon du c tand w oul d not c omment onmul ti ple occasion s  t his week o n w hen the admi nis trator  w oul d  retu rnto hi s comm i ttee.
“He wasju s t here earlier t hi s ye ar and answered q u es t ions for 2½ ho urs, b ut Ie x pec t h im to come  b ack a g ain,” B a rr a s s o t o ld reporters. 

 As you m a y h ave se en  in previo us bl og pos ts, we have  a W arboard i n ouro ffice w h ich sho ws the  cu rre nt s t at us  of  t h e s e rv ers wemanage.Mo st  of o ur c u s t o mers  ar e us ing o ur new alerti ngstac k , b ut so me ha ve t he ir ow n moni torin gsolutions which  wewant to i nte gr ate  w ith.O ne ofthe sewas Pro metheus .  Th is b l og  post covers how w et r an sformedraw Pr o me th e us value sint o pe r c entag es w h ic h we cou ld dis p la yon ourwa rboard  a ndcre ate a l erts  again st.

Int egrat i ng with Promet h eus

In order to g et a su mma ry of the d ataf romProm e th eus to di splay  o n  t h e Warboard we first  n eededto lo ok at what in formationt h e  Node Ex port e rp rovided and ho w itwas tagged .  Node Ex porte r is the  software w hich mak esthe  ra w s e r ve r stats avail a b le f or  Promet heu s to co lle ct.  Gi ven  t hat our p rimary c on ce rn sare  CPU, memory,di skIO  an d dis k  space  usag e we  n ee d edto cons t ru c t  que ri es  tocal cu latethem aspercent ages to b e d i spla ye don theWarboard.

Pr om etheus mak es  th e data a ccessibl e through its API on "/ a pi/v1/q uer y?que ry=< query>".Mos t of t hes yntaxis fai rly l og ica l witht h egen er al rule be i n g t hat i fwe t ake anaverage or maximum val ue we need to  sp ecif y "b y ( i ns t anc e) "  in o rder to keep ea ch serverse p ar ate. NodeE xpo rt er most ly retu rns r a w value s from th e kernelra ther t han  trying to  manipul ate them. Thisi sniceasit giv es  yo u fre edom t o  de cidehow to  u se the  data  b ut doesmean we have to give ou r quer ies a  li ttle extr a c o nsi deration: 

CP U

(1 - av g(i rat e(nod e_c p u{mod e="i d le "}[ 1 0m]))  by( instance) )  *  10 0 
C PU usage is b e ing  repo rte d  as an  int eger that increas e s over time so  wen e edtoc alcu la te the current  pe rc e n tage ofus age ourselv es. Fort un a telyPromet heus h as ther a te and ir atefun ctions fo r u s. Since r at ei s most ly f or  use i n calc u lati n g  whether or notalert t hresh o lds  have  b een cro s sedand we are  just tryin g to d ispl ay th e mostre cent d ata ,irate seem s  a better  fit.  We are  c urre ntl y taki ng d a t a ove r th elas t 10 minu t e st oen s ure  weget d at a fo r all
 at_work more_quickly.

EPW Chairman John Barrasso_(R-Wyo.) has_said_he planned_to_await the results_of a White_House review of Pruitt’s_conduct and would_not_comment on multiple occasions this week on when the administrator would return to his_committee.

“He_was just_here_earlier_this year and answered questions_for 2½ hours, but I_expect him_to come back again,” Barrasso told reporters.

 As_you_may have seen_in previous blog posts, we have a Warboard in_our office which shows the current_status of the_servers_we_manage. Most of our_customers are using our new alerting_stack, but some have their own_monitoring solutions which we want to integrate_with. One of these was Prometheus._This blog post covers how_we transformed_raw Prometheus values into percentages_which we could_display on_our warboard and_create alerts against.

Integrating with Prometheus

In order_to get a_summary of the data from Prometheus_to_display on the_Warboard_we_first needed_to look at_what_information the_Node_Exporter provided and how it was_tagged._Node Exporter is the software which makes_the raw server stats_available_for Prometheus to collect._Given that our primary concerns_are CPU, memory, disk IO and_disk space_usage we_needed to construct queries to calculate them as percentages to be_displayed on the Warboard.

Prometheus makes the_data accessible through its_API on_"/api/v1/query?query=<query>"._Most of the_syntax_is fairly_logical with the general rule being that_if we_take an average or maximum value_we need to specify_"by_(instance)" in order to keep each_server separate. Node Exporter mostly returns_raw values from the kernel_rather_than_trying to manipulate them. This_is nice as it gives you_freedom to decide_how to use the data but does_mean_we have to give our queries_a_little extra consideration:

CPU

(1 - avg(irate(node_cpu{mode="idle"}[10m])) by_(instance))_*_100

CPU usage is being reported_as an integer that increases over_time so we need to calculate the current percentage_of usage ourselves._Fortunately Prometheus has the rate_and_irate_functions for us. Since rate is mostly for use in_calculating whether_or not alert_thresholds have been crossed and we are just trying to_display the most recent data, irate seems_a better fit. We are currently taking data over the last_10 minutes to ensure we get data for_all