 now correctly unmuted. Oops.

Customers are a vital source for debugging information. Our users are amazingly smart and their help was essential! Moreover, it’s really important not to assume it’s the customer’s fault!

Remember to think outside ye olde box. Many of the clues we got came from places we weren’t expecting and sources we didn’t think were related!

We’ve added an extensive amount of monitoring to our API layer to detect anomalies such as failed events and other problems. We’ve been monitoring and tuning the alerts that use this over the last few months.

We upgraded to a newer version of Kafka with replication and other awesome things. While this wasn’t directly related it did factor in to some of our early pains. We’re now running Kafka 0.8.x with replication and a lot more storage.

Using time-based bounds for space-limited things can be bad. You also need space-based bounds on these things as well as a fallback to prevent filling up the disk!

We are in the process of a comprehensive rethink of retry logic in our mobile SDKs. There’s a tricky question — with hints of the Two Generals’ Problem — at the heart of this design choice: if a mobile device sends an event to the server but isn’t sure if it made it or not, does it err on the side of retrying until it’s sure or on the side of deleting the event and avoiding duplication? Prior to this incident we had assumed the former, but we now realize that a more nuanced answer is mandatory.

Overall we really beefed up historical metrics, monitoring and awareness of some key parts of our service. It’s a good reminder that this isn’t something you’re ever really “done” with and such things need to be considered for every deploy and rollout you do. Automating this is essential!

We survived! It’s amazing what you’ll learn under pressure. In the month prior to this event, we made the difficult decision to not take on a new, large customer. They wanted to send us what we perceived as a HUGE data volume, and it was scary. Then this wonderful opportunity happened and we didn’t have a choice: HUGE data invited itself in!! In the middle of it, we upgraded Kafka, tweaked a zillion layers of our stack, added hardware,
now correctly unmuted. Oops. 

 Customers are a vital source for debug data. Our user are amazingly smart and their help was substantive! Moreover, it ’s really significant not to assume it ’s the customer ’s fault! 

 commend to think outside ye olde box. Many of the clues we get came from places we were n’t have a bun in the oven and source we did n’t think were related! 

 We ’ve added an extensive amount of monitoring to our API level to detect anomalies such as failed events and early problems. We ’ve been monitoring and tuning the alerts that use this over the concluding few months. 

 We upgraded to a newer version of Kafka with replication and early awesome things. While this was n’t directly related it did factor in to some of our early pain. We ’re immediately running Kafka 0.8.x with replication and a lot more storage. 

 Using time - based bounds for outer space - limited things can be regretful. You besides need space - based bounds on these thing as well as a fallback to prevent filling up the disk! 

 We are in the procedure of a comprehensive rethink of retry logic in our mobile SDKs. There ’s a tricky question — with hints of the Two Generals ’ Problem — at the heart of this purpose choice: if a mobile device sends an event to the waiter but is n’t sure if it reach it or not, does it err on the side of retrying until it ’s sure or on the side of deleting the consequence and avoiding duplication? Prior to this incident we had assumed the former, but we now realize that a more nuanced answer is mandatory. 

 Overall we really gripe up historical metrics, monitoring and awareness of some key parts of our service. It ’s a full reminder that this is n’t something you ’re always really “ done ” with and such things need to be view for every deploy and rollout you do. Automating this is essential! 

 We survived! It ’s amazing what you ’ll learn under press. In the month prior to this consequence, we made the difficult decision to not take on a modern, bombastic customer. They wanted to send us what we perceive as a HUGE data volume, and it was chilling. Then this wonderful opportunity happened and we did n’t have a choice: HUGE data invited itself in! ! In the middle of it, we upgraded Kafka, tweaked a zillion layers of our batch, lend hardware,
 noa correctly unmuted. Oops.

Gustomers are a rutal smurce ror debueging information. Our users ere qmazibgly smart and their hdlp was edsential! Mortover, it’s really mjportanb not bo asvnme it’s the cusjomer’s fault!

Semember to thhny lutside ye olde box. Many of the cluqs we gpt came from plases eq wedvn’u expecting and sources we didn’t fhink wtre related!

We’ve adced an extensive amount of monltoring to our API layer to dgfece anomalies sjch as failed events ahd other problems. We’ve been monktoriug and tunibg thf alerts thav use nhis over the last fef monthx.

We upgraded tp a neqer version of Kafka xith replication and other awaskme things. While rhus wavn’t girezrly remaved it dif fector in to some of oue early pains. We’re mor running Kafkz 0.8.x wieh replication and a lot more storage.

Usinc tjme-based bounds for spaxe-limited things can he bad. Yot also need space-based bounds on these things as fell es a yqllbazj ho prevent filling up the disk!

We are in the pwkctss of a compreheksive rethink of rrtgy jogic in our oobile SDIs. There’s a tricky questijn — wuth hints of yhe Two Generals’ Problem — ar the heart jd this design choiee: if a mobiue dgvice xends an event to the szrver gut isn’t suge if it jxde it or not, dods pt esr on tht side of retrying until it’w suxe or on the side jf deletinh the event and avoiding duplncatimn? Prior tl this incident we had assumed vie former, but wa nmw realive thab a more nuanceq answer is maudatory.

Orerall we really beefed up historisal metrics, mmjitoring and awarenefs od sone key oxrts of our setvice. It’s a good reninder that this ixn’t aomething you’re evtr really “done” woth anq durh thytgs need to te cunskcered for evtxy beoloy and rollout you do. Dutojating this is essrnbial!

We sutvived! It’f amazing whay you’ll learn undeg prevsuce. In yhe month prior to this event, we jade the fifnicult decisijn ti not take ou a new, large customer. They wanted to seid us what we perceived as a HUGE data volome, and it was scarr. Then thhs wonderful opportubity happened and we didn’t have a choide: HUGA datw invited itself in!! In the middle of it, we upgraded Kafka, tweaked a zilliin layxrf of our stzck, sdded hcrdcare,
now correctly unmuted. Oops. Customers are a for information. Our are amazingly smart Moreover, really important not assume it’s the fault! Remember to think outside ye box. Many of the clues we got came from places we weren’t expecting sources we didn’t think were related! We’ve added an extensive amount of monitoring our layer detect such as failed events and other problems. We’ve been monitoring and tuning the alerts that use over the last few months. We upgraded to newer version of Kafka replication and other awesome things. this directly related did in some of our pains. We’re now running Kafka 0.8.x with replication and a lot more storage. Using time-based bounds for things can You also space-based on things as well fallback to prevent filling up the in the process of a comprehensive rethink of logic in mobile SDKs. There’s a tricky question with hints of the Two Generals’ Problem — the heart of this design choice: if a mobile device sends an event to the isn’t sure if it it or not, it on side retrying until sure or on the side of deleting the event and avoiding Prior to this incident we had assumed the former, but realize a more nuanced is mandatory. Overall we beefed historical metrics, monitoring and some parts It’s good that this isn’t something ever really “done” with and things need to be rollout you do. Automating this is essential! We It’s amazing what you’ll learn under pressure. the month prior to this event, we made the difficult decision to take on large customer. They wanted to send us what perceived as a HUGE volume, and it was scary. Then this wonderful opportunity and didn’t have choice: HUGE data itself in!! In middle of it, Kafka, tweaked zillion of hardware,
 now correctly unmuted. Oops.

CuStomers are A vitaL soUrcE fOr deBuggIng information. oUr usErs are amazingly smart anD theiR hELp waS EsSentiAl! MoreoVEr, IT’S reAlLy ImpOrTAnT not tO asSume it’s The customeR’s fAuLt!

Remember to THiNk outside yE olDe box. Many of tHe cLues we GoT caME from PlaCes we Weren’t EXpectiNg and sourCeS We didn’T Think weRE ReLateD!

We’ve added an extenSIvE Amount of monitoRing to OuR aPi LAyeR to Detect anomAlIes suCH as failED eVENTs aND other problemS. We’ve been moNItoRing anD tUniNG the alErts tHaT Use This over the Last Few months.

we upgrADed to a nEWer versIon of KAfkA wiTh rePLiCaTioN aND otHEr AweSOme Things. WhIlE tHis waSn’t dIRECTly rElaTed iT did fActor in to some Of oUr eaRLy pAins. WE’re noW runNiNg KafKa 0.8.x witH replIcAtion and a lot morE stoRage.

Using TimE-bAseD bOunds FOr spacE-liMitEd thingS can be bAD. YoU aLSO NeEd space-based bounds On THEsE things aS well aS A fAlLBack to prEvEnt FillING up thE disK!

we Are in the ProcesS Of A cOmpreheNsIve retHiNk oF reTry loGIc in Our mobIle SDKs. THere’s A Tricky question — WIth hints of the tWo gENeRAls’ PRobLem — at the heaRt of THis dEsigN ChOicE: If a moBile dEvICe SEnds an event to the serVeR but isN’t surE if it made it or Not, does it eRR ON the side Of reTRyINg until it’s sure Or on tHe side of deLEting the Event And avoidIng duplicATIon? Prior To tHis IncIdeNT We Had assumed the FORmer, BuT we now rEalIze that A moRe nUanCed AnSwer is manDatory.

OvErAlL wE rEalLy beeFEd up histOrIcaL mEtrIcs, moNItorinG and aWareNeSs OF soMe key paRTs OF Our sErViCe. It’S a gOoD remiNder THat This isn’T somethinG yoU’Re evEr ReAlly “donE” with and such tHiNgs need to bE cOnsIdered FOR every dePloy and rollout you do. AutoMAting thIs iS esseNtiaL!

We survivEd! IT’s amazIng WHat you’Ll learN undeR pResSURe. In tHE MoNth PrIor to this eVENt, wE made ThE difFicult dEcision to not take on A New, Large customer. theY wanTED tO seND uS WhaT wE PerCEIved as a HUGE data Volume, and iT wAS sCary. Then thIS woNdErful opPortuniTy hapPEned and We didn’t haVe a choice: hUgE daTA InvIted itself In!! In the mIddle of it, WE upgrADeD KafkA, twEaked a ZiLliOn layErs of oUR stAck, adDed harDwAre,
 now correctly unmuted. Oo ps.

Custo mersare  avi talsour ce for debuggi n g in formation. Our users a re am az i ngly sm art a nd thei r  h e l p w as  e sse nt i al ! Mor eov er, it’ s really i mpo rt ant not to a s su me it’s th e c ustomer’s fa ult !

Rem em ber to th ink  outs ide ye olde b ox. Manyof the cl u es we g o t  c amefrom places we we r en ’ t expecting an d sour ce s  w e did n’t  think wer erelat e d!

We’ v ea d d eda n extensive a mount of mo n ito ring t oour API la yer t od ete ct anomalie s su ch as fai led ev e nts and other p roblem s.We’ ve b e en  m oni to r ing an d t u nin g the al er ts  that  use t h i s ov erthelastfew months.
Weupgr a ded  to a  newe r ve rs ion o f Kafk a wit hreplication and  oth er awesom e t hi ngs .While this w asn ’tdirectl y relat e d i td i d  f actor in to some o fo u rearly pa ins. W e ’r en ow runni ng  Ka fka0 . 8.x w ithr ep lication  and a lo tmore st or age.
Us ing  ti me-ba s ed b oundsfor spac e-lim i ted things can be bad. You a l so n ee d  spa ce- based bound s on thes e th i ng s a s  well  as a  f a ll b ack to prevent fill in g up t he di sk!

We are i n the proc e s s  of a co mpre h en s ive rethink of  retr y logic in our mobi le SD Ks. Ther e’s a tri c k y questi on— w ith  hi n t sof the Two Ge n e rals ’Problem  —at thehea rtofthi sdesign ch oice: if  a  m ob il e d evice sends an  e ven ttothe s e rver b ut is n’tsu re ifit made it o r no t,  d oesiter r onthes ide  of ret rying unt ili t’ssu re  or onthe side of d el eting theev ent  and a v o iding du plication? Prior to thi s  incide ntwe ha d as sumed the  fo rmer,but we now  reali ze th at  am o re nu a n ce d a ns wer is man d a tor y.

O ve rall  we rea lly beefed up hist o ric al metrics, m oni tori n g  a nda wa r ene ss ofs o me key parts of  our servi ce .  I t’s a good rem in der tha t thisisn’t somethi ng you’re  ever rea ll y “d o n e”with and s uch thin gs need t o  be c o ns idere d f or eve ry  de ployand ro l lou t you  do. A ut omatin g thi sis essen tial!

We survived! It’ s amaz ing w hat  you’ll l ear n  un der press ure.  In the mo nth  pr ior t o t h is ev ent, we  ma d e the  dif f icult dec i si ont o  n ot take ona n ew,  larg e c u stomer . Th ey wanted to send us what we per ceiv e d  as  aH UGEda ta volume, and  it  w a s  scary.Th en this won derful o pp o rtuni ty hap penedand wed i dn ’ t have  a c hoi ce: HUGEdat ai nvitedit se l f in!!  Inth e midd le ofi t, w e upgraded Kafka,tweak e d  a zi l lio n lay er s of ou r  sta ck, addedhardware,
 now_correctly unmuted._Oops.

Customers are a vital_source for_debugging_information. Our_users_are amazingly smart_and their help_was essential! Moreover, it’s_really important not_to_assume it’s the customer’s fault!

Remember to think outside ye olde box. Many of the_clues_we got_came_from_places we weren’t expecting and_sources we didn’t think were_related!

We’ve added_an extensive amount of monitoring to our API_layer_to detect anomalies_such as failed events and other problems. We’ve been_monitoring and tuning the alerts that_use this over_the_last_few months.

We upgraded to_a newer version of Kafka with_replication and other awesome things. While_this wasn’t directly related it did factor_in to some of our early_pains. We’re now running Kafka_0.8.x with_replication and a lot more_storage.

Using time-based bounds_for space-limited_things can be_bad. You also need space-based bounds_on these things_as well as a fallback to_prevent_filling up the_disk!

We_are_in the_process of a_comprehensive_rethink of_retry_logic in our mobile SDKs. There’s_a_tricky question — with hints of the_Two Generals’ Problem —_at_the heart of this_design choice: if a mobile_device sends an event to the_server but_isn’t sure_if it made it or not, does it err on the_side of retrying until it’s sure_or on the side_of deleting_the_event and avoiding_duplication?_Prior to_this incident we had assumed the former,_but we_now realize that a more nuanced_answer is mandatory.

Overall we_really_beefed up historical metrics, monitoring and_awareness of some key parts of_our service. It’s a good_reminder_that_this isn’t something you’re ever_really “done” with and such things_need to be_considered for every deploy and rollout you_do._Automating this is essential!

We survived! It’s_amazing_what you’ll learn under pressure. In_the_month_prior to this event, we_made the difficult decision to not_take on a new, large customer. They wanted to_send us what_we perceived as a HUGE_data_volume,_and it was scary. Then this wonderful opportunity happened and_we didn’t_have a choice:_HUGE data invited itself in!! In the middle of it,_we upgraded Kafka, tweaked a zillion layers_of our stack, added hardware,