 configuration to ensure failovers will be instantaneous going forward. We have ensured that failovers for this cluster may only be initiated during rare, scheduled downtime, when there will be no impact on customers.

June 26th

From 13:37 PT to 14:09 PT (20:37–21:09 UTC), Coinbase.com experienced sustained error rates across all endpoints. Error rates hovered around 35% for the duration of the incident.

This moderate service degradation was caused by sustained increased query latency on a MongoDB cluster that stores user account data. There were several contributing factors that led to increased query latency and degraded service:

Before the incident began, a background job performed an aggregation for a large number of accounts, causing excessive reads into cache and cache evictions for the cluster’s storage engine cache. Soon after, a large number of real-time price alerts were triggered, causing significantly increased query throughput for this cluster. Increased query throughput, combined with the prior cache evictions, caused additional cache pressure and resulted in query queueing and increased query latency. These are initial conclusions; in collaboration with MongoDB our investigation is ongoing. Due to increased query latency during the request/response cycle, web workers became saturated, serving HTTP 502s.

Storage engine cache activity for the affected cluster

The incident was resolved when we manually failed over the affected cluster to instances with more memory. Since the incident, we’ve done the following:

We’ve audited working set size across all clusters, resizing instances and/or reducing the working set size as necessary.

We’ve removed the background job and are performing an audit of similar queries, moving them to analytics nodes.

We’ve tuned certain high-throughput endpoints that are hit hardest during price alerts.

We’re continuing ongoing work to reduce load on MongoDB through caching and reading from secondary data stores that can be scaled horizontally.

We take uptime very seriously, and we’re working hard to support the millions of customers that choose Coinbase to manage their cryptocurrency. If you’re interested in solving scaling challenges like those presented here, come work with us.

Unless otherwise noted, all images provided herein are by Coinbase. 947 SHARES Facebook Twitter Whatsapp Pinterest Reddit Pocket

ALEC: learn this acronym well. It stands for American Legislative Exchange Council.

ALEC is classified as a a 501(c)(3) company,
configuration to ensure failovers will be instantaneous going forward. We have guarantee that failovers for this bunch may only be initiated during rare, schedule downtime, when there will be no shock on customers. 

 June 26th 

 From 13:37 PT to 14:09 PT (20:37–21:09 UTC), Coinbase.com experienced free burning error rates across all endpoints. mistake rates hovered around 35% for the duration of the incident. 

 This moderate serve degradation was caused by sustain increase query latency on a MongoDB cluster that stores user history data. There were several put up factors that led to increased query rotational latency and degraded service: 

 Before the incident began, a background job perform an aggregation for a large number of accounts, causing excessive reads into cache and cache evictions for the cluster ’s storage engine cache. Soon after, a large number of real - clock time price alerts were trigger, cause significantly increased query throughput for this bunch. Increased question throughput, combined with the prior cache eviction, caused additional cache pressure and resulted in query queueing and increased query latency. These are initial conclusions; in collaboration with MongoDB our probe is ongoing. Due to increased question latency during the request / response cycle, web workers became saturated, serve HTTP 502s. 

 Storage engine cache natural process for the affected cluster 

 The incident was resolved when we manually failed over the feign cluster to example with more memory. Since the incident, we ’ve done the following: 

 We ’ve audit working sic size across all clusters, resize instances and/or reducing the work set size as necessary. 

 We ’ve removed the backdrop job and are do an audit of similar queries, moving them to analytics nodes. 

 We ’ve tuned certain high - throughput endpoints that are reach hardest during price alert. 

 We ’re continuing ongoing work to reduce load on MongoDB through caching and reading from junior-grade data stores that can be scaled horizontally. 

 We take uptime very seriously, and we ’re working hard to support the million of customer that choose Coinbase to wangle their cryptocurrency. If you ’re interested in solving scaling challenges like those presented here, come employment with us. 

 Unless otherwise noted, all images put up herein are by Coinbase. 947 SHARES Facebook Twitter Whatsapp Pinterest Reddit Pocket 

 ALEC: learn this acronym well. It stands for American Legislative Exchange Council. 

 ALEC is classified as a a 501(c)(3) company,
 cojfiguration to ensure fallovers will be nbstanteneous foing fofward. We have ensured that feilocers dor this cluster may ovly be inptiated dyrinj rare, scheduled downtimc, wheh thexe will be no imkact on custmmers.

June 26th

Frmm 13:37 'T to 14:09 PT (20:37–21:09 UTC), Coinbase.com experiencqd sustsijed error ratef acgofs aml endpoints. Error rates hovered adound 35% hor the duratiom of the incident.

This modegate service degradatiln was causgs br sustained ivcreased qlzry latency on a MongoDB cluster that stords uszr account eara. Hvere were sxveral contributinn factoss that led to increaxed quwry latency and degraved service:

Before thg incident bzgan, a background job pwrforked dn aetreeatjoi fkr a lwrgx number of accounts, cqusing excessive resdf into cache ahd cacre evictions for the cluster’s storage engpne dache. Soon after, a largw number of real-time krice aleres were triggered, causing significantly increased querb ghriunhpug flr this cluster. Increased query throughput, coigimec with the prijr cache evochipgs, caused addktional czche pressure and gesulteq in wuery queteinb and increased query latenxy. These are unitial conclusiond; in collaburatoon woth MongoDB our investiyation is ongoing. Due to ihzreased query lagenby dgring tht request/response sycle, web worlers bezame saturwted, servijg HTB[ 502s.

Storage engine fache awtivity fog the affected cluster

The incidxit was resolvgd fhet we manbally nailed over the affected clusjer to inftancds with moge memory. Since the yncident, we’ve fone the folnowing:

We’de ayditwd workkvg set size actoss all blbsters, rewizing instances akd/or tesucing the worknug set size as nevesrarr.

Wv’ve rqkoved the bawkgruuna job xnd are pernoroing an audit of similar quedies, moving them tp wnalyticw nodes.

Wq’ve tuned ceryain high-throughpuu endpminvs thay ate hit hardest during price aledts.

We’re clntlnuing ongoind wovk tj reduce lpad on MongoDB through caching and readmng from secondary data stores that can be xcaled horivontaljy.

We take uptime very seriousoy, and we’re workikg hard to support the millimns ov customers that choose Coinbase to manage their cryptocurrency. If you’re unterested in solvjng xcalitg ehallendes pmke those presentvd here, come work with us.

Unless utherwisa uoted, all images provided herriv are by Coincase. 947 SHARES Facebook Twjtter Whstsapp Pinterest Reddit Pocket

AKEC: learn this acronym wcll. It stqnds ror Amvticen Legislative Exchamge Cobncil.

AOEC is xlasdified as a a 501(c)(3) volpany,
configuration to ensure failovers will be instantaneous We ensured that for this cluster rare, downtime, when there be no impact customers. June 26th From 13:37 PT 14:09 PT (20:37–21:09 UTC), Coinbase.com experienced sustained error rates across all endpoints. Error hovered around 35% for the duration of the incident. This moderate service degradation caused sustained query on a MongoDB cluster that stores user account data. There were several contributing factors that led increased query latency and degraded service: Before the began, a background job an aggregation for a large of causing excessive into and evictions for the storage engine cache. Soon after, a large number of real-time price alerts were triggered, causing significantly increased throughput for Increased query combined the cache evictions, caused pressure and resulted in query queueing latency. These are initial conclusions; in collaboration with our investigation ongoing. Due to increased query latency the request/response cycle, web workers became saturated, serving 502s. Storage engine cache activity for the affected cluster The incident was resolved when we over the affected cluster instances with more Since incident, done following: We’ve working set size across all clusters, resizing instances and/or reducing the set size as necessary. We’ve removed the background job and an of similar queries, them to analytics nodes. tuned high-throughput endpoints that are during alerts. work reduce on MongoDB through caching reading from secondary data stores can be scaled horizontally. and we’re working hard to support the millions customers that choose Coinbase to manage their If you’re interested in solving scaling challenges like those presented here, come with us. noted, all images provided herein are by Coinbase. SHARES Facebook Twitter Whatsapp Reddit Pocket ALEC: learn this acronym well. It stands American Exchange Council. is classified as a 501(c)(3) company,
 configuration to ensure failOvers will bE instAntAneOuS goiNg foRward. We have ensURed tHat failovers for this cluSter mAy ONly bE InItiatEd durinG RaRE, SchEdUlEd dOwNTiMe, wheN thEre will Be no impact On cUsTomers.

June 26th

fRoM 13:37 PT to 14:09 PT (20:37–21:09 UTC), coiNbase.com expeRieNced suStAinED erroR raTes acRoss alL EndpoiNts. Error rAtES hoverED around 35% FOR tHe duRation of the incideNT.

THIs moderate servIce degRaDAtION waS caUsed by sustAiNed inCReased qUErY LATenCY on a MongoDB clUster that stORes User acCoUnt DAta. TheRe werE sEVerAl contributIng fActors thaT led to INcreaseD Query laTency aNd dEgrAded SErViCe:

BEfORe tHE iNciDEnt Began, a baCkGrOund jOb peRFORMed aN agGregAtion For a large numbEr oF accOUntS, causIng exCessIvE readS into cAche aNd Cache evictions fOr thE cluster’s StoRaGe eNgIne caCHe. Soon AftEr, a Large nuMber of rEAl-tImE PRIcE alerts were triggerEd, CAUsIng signiFicantLY iNcREased queRy ThrOughPUT for tHis cLUsTer. IncreAsed quERy ThRoughpuT, cOmbineD wIth The Prior CAche EvictiOns, causeD addiTIonal cache presSUre and resulteD In QUErY QueuEinG and increasEd quERy laTencY. thEse ARe iniTial cOnCLuSIons; in collaboration WiTh MongODB ouR investigatioN is ongoing. dUE To increaSed qUErY Latency during tHe reqUest/responSE cycle, weB workErs becamE saturateD, SErving HTtP 502s.

stoRagE enGINe Cache activity FOR the AfFected cLusTer

The iNciDenT waS reSoLved when wE manuallY fAiLeD oVer The afFEcted cluStEr tO iNstAnces WIth morE memoRy. SiNcE tHE inCident, wE’Ve DONe thE fOlLowiNg:

WE’vE audiTed wORkiNg set siZe across aLl cLUsteRs, ReSizing iNstances and/or ReDucing the wOrKinG set siZE As necessAry.

We’ve removed the backgrOUnd job aNd aRe perFormIng an audiT of SimilaR quERies, moVing thEm to aNaLytICS nodeS.

wE’vE tuNeD certain hiGH-ThrOughpUt EndpOints thAt are hit hardest durINg pRice alerts.

We’rE coNtinUINg OngOInG WorK tO RedUCE load on MongoDB tHrough cachInG AnD reading frOM seCoNdary daTa storeS that CAn be scaLed horizoNtally.

We tAkE uptIME veRy seriouslY, and we’re Working haRD to suPPoRt the MilLions oF cUstOmers That chOOse coinbAse to mAnAge theIr cryPtOcurrencY. If you’re interested in solVing scAling ChaLlenges liKe tHOse Presented Here, Come work wiTh uS.

UnLess oTheRWise nOted, ALl ImaGEs proVideD Herein are BY COinBASe. 947 sHARES FacebOOK twiTter WHatSApp PinTereSt Reddit Pocket

ALEc: Learn this acronYm weLL. it sTanDS for amErican LegislatIve exCHAnge CounCiL.

ALEC is clasSified as A a 501(C)(3) CompaNy,
 configuration to ensure f ailovers w ill b e i nst an tane ousgoing forward. We h ave ensured that failo versfo r  thi s  c luste r may o n ly b e i ni ti ate dd ur ing r are , sched uled downt ime ,when there w i ll  be no imp act  on customer s.
June26 thFrom13: 37 PT  to 14 : 09 PT(20:37–21 :0 9  UTC), Coinbas e . co m ex perienced sustain e de rror rates acr oss al le nd p o int s.Error rate shover e d aroun d  3 5 % for the durationof the inci d ent .

Thi smod e rate s ervic ed egr adation was  cau sed by su staine d  increa s ed quer y late ncy  on  a M o ng oD B c lu s ter th ats tor es userac co unt d ata. T h e re w ere  sev eralcontributingfac tors tha t led  to i ncre as ed qu ery la tency  a nd degraded ser vice :

Before  th einc id ent b e gan, a  ba ckg round j ob perf o rme da n ag gregation for a la rg e nu mber ofaccoun t s,  c a using ex ce ssi ve r e a ds in to c a ch e and ca che ev i ct io ns forth e clus te r’s  st orage engi ne cac he. Soon  afte r , a large numb e r of real-tim e  p r i ce aler tswere trigge red, caus ings ig nif i cantl y inc re a se d  query throughput f or  thisclust er. Increased  query thr o u g hput, co mbin e dw ith the priorcache  evictions ,  causedaddit ional ca che press u r e and re sul ted  in  qu e r yqueueing andi n crea se d query  la tency.The seare  in it ial concl usions;in  c ol la bor ation with Mon go DBou r i nvest i gation  is o ngoi ng .D ueto incr e as e d  que ry  l aten cydu ringther equ est/res ponse cyc le, webwo rk ers bec ame saturated ,serving HT TP  50 2s.

S t o rage eng ine cache activity fort he affe cte d clu ster 

The inc ide nt was  re s olvedwhen w e man ua lly f ailed o ve r t he  affectedc l ust er to  i nsta nces wi th more memory. Si n cethe incident,  we ’ved o ne  th e  f o llo wi n g:W e’ve audited wo rking setsi z eacross all clu st ers, re sizinginsta n ces and /or reduc ing the w or king s etsize as ne cessary. 

We’ve r e moved th e bac kgr ound j ob  an d are  perfo r min g anauditof  simil ar qu er ies, mov ing them to analytics n odes.
We’v e t uned cert ain hig h-through putendpointstha t a re hi t h a rdest  dur i ng  pr i ce al erts . 

We’re c o nt inu i n gongoing wor k t o r educe  lo a d on M ongo DB through cachin g  and reading f roms e con dar y  dat astores that ca n b es c aled hor iz ontally.

W e take u pt i me ve ry ser iously , and w e ’ re workin g ha rdto suppor t t he million sof custom ersth at cho ose Co i nbas e to manage theircrypt o c urren c y.If yo u’ re inte r este d in solvi ng scalingchalle nges  like  thosepr esente d h er e, come wo r k with us .

Un less ot he rwis e n oted,alli m agesprov id edherein ar e by Co in b ase . 94 7 SHA RE S Fa cebook Tw i tter Wha tsa p p Pinte re stR e ddit P o ck e t 

ALEC: le arn  this a cronym wel l . It st a nds f or Ame ricanLegisla t ive  E xchange  Co u n cil.

ALE C is clas s ifi ed  asa a 501( c) (3)co mpa ny ,
 configuration_to ensure_failovers will be instantaneous_going forward._We_have ensured_that_failovers for this_cluster may only_be initiated during rare,_scheduled downtime, when_there_will be no impact on customers.

June 26th

From 13:37 PT to 14:09 PT (20:37–21:09 UTC),_Coinbase.com_experienced sustained_error_rates_across all endpoints. Error rates_hovered around 35% for the_duration of_the incident.

This moderate service degradation was caused by_sustained_increased query latency_on a MongoDB cluster that stores user account data._There were several contributing factors that_led to increased_query_latency_and degraded service:

Before the_incident began, a background job performed_an aggregation for a large number_of accounts, causing excessive reads into cache_and cache evictions for the cluster’s_storage engine cache. Soon after,_a large_number of real-time price alerts_were triggered, causing_significantly increased_query throughput for_this cluster. Increased query throughput, combined_with the prior_cache evictions, caused additional cache pressure_and_resulted in query_queueing_and_increased query_latency. These are_initial_conclusions; in_collaboration_with MongoDB our investigation is ongoing._Due_to increased query latency during the request/response_cycle, web workers became_saturated,_serving HTTP 502s.

Storage engine_cache activity for the affected_cluster

The incident was resolved when we_manually failed_over the_affected cluster to instances with more memory. Since the incident, we’ve_done the following:

We’ve audited working set_size across all clusters,_resizing instances_and/or_reducing the working_set_size as_necessary.

We’ve removed the background job and are_performing an_audit of similar queries, moving them_to analytics nodes.

We’ve tuned_certain_high-throughput endpoints that are hit hardest_during price alerts.

We’re continuing ongoing work_to reduce load on MongoDB_through_caching_and reading from secondary data_stores that can be scaled horizontally.

We_take uptime very_seriously, and we’re working hard to support_the_millions of customers that choose Coinbase_to_manage their cryptocurrency. If you’re interested_in_solving_scaling challenges like those presented_here, come work with us.

Unless otherwise_noted, all images provided herein are by Coinbase. 947_SHARES Facebook Twitter_Whatsapp Pinterest Reddit Pocket

ALEC: learn_this_acronym_well. It stands for American Legislative Exchange Council.

ALEC is classified_as a_a 501(c)(3) company,