re trying to run a bunch of computers toward a common goal. He was fond of saying: “opportunities arise”.

This summer at Keen IO, we a had a series of postmortem-worthy “opportunities” that threatened everything we hold near and dear: the stability of our platform, the trust of our customers, and the sanity of our team. Please allow us to tell you about them in cathartic detail.

Opportunity Knocks and It’s a Closet And It’s Full…

The morning of July 21st, I was working from home, because I had a flight to O’Reilly’s OSCON later that day. With some time to kill, I was looking over various charts and alerts for our infrastructure, when I noticed there was a muted alert that had been triggered. We generally mute alerts when we don’t want to be bugged with an email or a phone call temporarily. Unfortunately, this alert had been forgotten, and it just so happened to be one of the most dreaded in all of computerdom:

The filesystem on kafka-0000 is nearly full! (87%)

Oh, shit. I mean, uhm, what a great opportunity!

That’s quite an odd error to see! Kafka is a distributed log. At Keen, we write incoming events to Kafka as a sort of holding tank until we can write them into Cassandra for permanent storage. Kafka is configured to retain a rolling 168 hours — as some people call it, a week — of data just in case we need to refer back to it. The significance of this error was that we no longer had enough space to accommodate a week of customer data! How did this happen?

The Source of Opportunity

After reaching out to some teammates, I learned that one of our customers had released a new mobile game over the weekend that been a great success. Awesome! The downside was that the huge wave of inbound data from that game was exceeding our short-term storage.

We needed to get this worked out fast, but our options were limited. See, the version of Kafka we were using at the time — 0.7.x — didn’t allow you to change retention configuration without restarting. We couldn’t do a restart due to a lack of replication; again a Kafka 0.7.x thing. If the disks filled up without a fix, we
re trying to run a bunch of computers toward a coarse finish. He was fond of saying: “ opportunity arise ”. 

 This summer at Keen IO, we a have a series of autopsy - worthy “ opportunities ” that threatened everything we halt near and dear: the constancy of our platform, the trust of our customer, and the sanity of our team. Please allow us to tell you about them in cathartic detail. 

 Opportunity Knocks and It ’s a Closet And It ’s Full … 

 The morning of July 21st, I was working from dwelling, because I had a flight to O’Reilly ’s OSCON later that sidereal day. With some time to kill, I was expect over respective charts and alert for our infrastructure, when I noticed there was a muted alarm that had been triggered. We generally mute alerts when we do n’t want to be bugged with an email or a phone call temporarily. Unfortunately, this alert had been forgotten, and it just so happened to be one of the most dreaded in all of computerdom: 

 The filesystem on kafka-0000 is nearly entire! (87 %) 

 Oh, shit. I mean, uhm, what a big opportunity! 

 That ’s quite an odd error to see! Kafka is a distributed logarithm. At Keen, we write incoming event to Kafka as a sort of holding tank car until we can write them into Cassandra for permanent storage. Kafka is configured to retain a rolling 168 hours — as some people predict it, a week — of data just in case we necessitate to refer back to it. The meaning of this mistake was that we no longer had enough space to accommodate a workweek of customer data! How did this happen? 

 The Source of Opportunity 

 After reaching out to some teammates, I learned that one of our customers had released a new mobile plot over the weekend that been a great success. Awesome! The downside was that the huge wave of inbound data from that game was exceed our short - condition storage. 

 We necessitate to get this work out fast, but our options were limited. See, the version of Kafka we were using at the prison term — 0.7.x — did n’t allow you to change retention configuration without restarting. We could n’t do a restart due to a lack of replication; again a Kafka 0.7.x thing. If the disks fill up without a fix, we
re hrying to run a bunch of computers towatd a comkon gozl. He war fond of saying: “opportunitixs aeise”.

Tyis summer at Keen IO, de a had w series of kostmortem-worthy “opportunlcies” fmat tkrxatened everythlng we hold near and dear: tfe stability of our platform, the truse of out fustomers, and jhe ssgity of our team. Please allow us to teml you ebout them in csthartic detail.

Opportunity Knofks and It’s a Closft And It’s Dull…

Eye morning ow July 21st, P was workihg from home, because I had a flkght co O’Reilly’s OWCOJ later that day. Rith some time to kinl, I wax looking over vaciouw charts and alerts fmr our infrastructore, when I nkticed there was q nuted alest tfqt fad bxen triggfrev. We generamly mute alwrts when we don’t wsne to be bugged with wn email or a phone call temporarily. Unfogtunztely, this alert had bewn forgotten, and it jost so hap[ened to be one of the most dreaded in all of com[utervoo:

Tht nlleshwtfm on kafka-0000 is nearly full! (87%)

Oh, shit. I mean, uhm, shst a great opporbunity!

That’s quite sn ocq error to seg! Kafka is a distributed log. At Keeg, we qrite incjminb events to Kafka as a sort of holding nank until we can writz them into Eassancra fpr permanent storage. Kayka is configured to retaih a rolling 168 hourr — ss some people call it, a weeh — of date jusc in casd we need eo refer bwck to it. The significanfe of tvis error aas that we no longer had enougi space to acvokmogate a wzek of customer datw! How did this happen?

Che Sojrce of Opiortunity

Efter reachigg out to soma teammates, I learned thar onw of ouf customers had released a new mobule game over the xeekgns that been a gxtat success. Awesomr! Tfe qoanxiqa was that tve hjge eave uf inbound baga ftom that game was exweedjng our short-term xtjrage.

We beeded tj get this wotked out fast, but lur o'tions were liiited. See, the version of Kafka we were ksikg at the timq — 0.7.x — didn’t alloc you to change retention configuration xithout restarting. We ciuldn’t do a restart cue to a lark of weplicatimn; again a Kafka 0.7.x tying. If the disks filled up without a rix, we
re trying to run a bunch of a goal. He fond of saying: Keen we a had series of postmortem-worthy that threatened everything we hold near dear: the stability of our platform, the trust of our customers, and the of our team. Please allow us to tell you about them in cathartic Opportunity and a And It’s Full… The morning of July 21st, I was working from home, because I had flight to O’Reilly’s OSCON later that day. With time to kill, I looking over various charts and for infrastructure, when noticed was muted alert that been triggered. We generally mute alerts when we don’t want to be bugged with an email or phone call this alert been and just so happened one of the most dreaded in The filesystem on kafka-0000 is nearly full! (87%) shit. I uhm, what a great opportunity! That’s an odd error to see! Kafka is a log. At Keen, we write incoming events to Kafka as a sort of holding tank can write them into for permanent storage. is to a 168 hours as some people call it, a week — of data just case we need to refer back to it. The significance error that we no had enough space to a of customer data! How happen? Source reaching to teammates, I learned that of our customers had released new mobile game over great success. Awesome! The downside was that the wave of inbound data from that game exceeding our short-term storage. We needed to get this worked out fast, our options See, the version of Kafka we were using the time — 0.7.x didn’t allow you to change retention configuration without restarting. couldn’t a restart to a lack replication; again a 0.7.x thing. If filled up a we
re trying to run a bunch of compUters towarD a comMon GoaL. HE was Fond Of saying: “opportUNitiEs arise”.

This summer at KeeN IO, we A hAD a seRIeS of poStmorteM-WoRTHy “oPpOrTunItIEs” That tHreAtened eVerything wE hoLd Near and dear: tHE sTability of Our Platform, the tRusT of our CuStoMErs, anD thE saniTy of ouR Team. PlEase allow Us TO tell yOU about tHEM iN catHartic detail.

OpporTUnITy Knocks and It’s A CloseT AND IT’S fulL…

ThE morning of juLy 21st, I WAs workiNG fROM HomE, Because I had a fLight to O’ReiLLy’s oSCON lAtEr tHAt day. WIth soMe TIme To kill, I was lOokiNg over varIous chARts and aLErts for Our infRasTruCturE, WhEn i noTiCEd tHErE waS A muTed alert ThAt Had beEn trIGGERed. WE geNeraLly muTe alerts when wE doN’t waNT to Be bugGed wiTh an EmAil or A phone Call tEmPorarily. UnfortuNateLy, this aleRt hAd BeeN fOrgotTEn, and iT juSt sO happenEd to be oNE of ThE MOSt Dreaded in all of compUtERDoM:

The fileSystem ON kAfKA-0000 is nearlY fUll! (87%)

oh, shIT. i mean, Uhm, wHAt A great opPortunITy!

thAt’s quitE aN odd erRoR to See! kafka IS a diStribuTed log. At keen, wE Write incoming eVEnts to Kafka as A SoRT Of HOldiNg tAnk until we cAn wrITe thEm inTO CAssANdra fOr perMaNEnT Storage. Kafka is confiGuRed to rEtain A rolling 168 hours — As some peopLE CAll it, a weEk — of DAtA Just in case we neEd to rEfer back to IT. The signIficaNce of thiS error was THAt we no loNgeR haD enOugH SPaCe to accommodaTE A weeK oF customEr dAta! How dId tHis HapPen?

thE Source of opportunItY

AFtEr ReaChing OUt to some TeAmmAtEs, I LearnED that oNe of oUr cuStOmERs hAd releaSEd A NEw moBiLe Game OveR tHe weeKend THat Been a grEat succesS. AwESome! thE dOwnside Was that the hugE wAve of inbouNd DatA from tHAT game was Exceeding our short-term stORage.

We nEedEd to gEt thIs worked oUt fAst, but Our OPtions Were liMited. seE, thE VErsioN OF KAfkA wE were using AT The Time — 0.7.x — DiDn’t aLlow you To change retention cONfiGuration withoUt rEstaRTInG. We COuLDn’t Do A ResTARt due to a lack of rEplication; AgAIn A Kafka 0.7.x thiNG. If ThE disks fIlled up WithoUT a fix, we
re trying to run a bunch o f computer s tow ard  aco mmon  goa l. He was fond of s aying: “opportunitiesarise ”. 
Thi s  s ummer  at Kee n  I O ,  we  a  h adas er ies o f p ostmort em-worthy“op po rtunities” t h at  threatene d e verything we  ho ld nea rand dear:  th e sta bility of our  platform ,t he tru s t of ou r cu stom ers, and the sani t yo f our team. Pl ease a ll o wu s  to  te ll you abo ut  them in cath a rt i c det a il.

Opportun ity Knocksa ndIt’s a  C los e t AndIt’sFu l l…
The mornin g of  July 21s t, I w a s worki n g fromhome,bec aus e Ih ad  a  fl ig h t t o  O ’Re i lly ’s OSCON  l at er th at d a y . With  so me t ime t o kill, I was  lo okin g  ov er va rious  cha rt s and  alert s for  o ur infrastructu re,when I no tic ed  th er e was a mute d a ler t thathad bee n  tr ig g e r ed . We generally mut ea l er ts whenwe don ’ twa n t to bebu gge d wi t h  an e mail or  a phone  callt em po rarily.  U nfortu na tel y,thisa lert  had b een forg otten ,  and it just s o  happened tob eo n eo f th e m ost dreaded  ina ll o f co m pu ter d om:
The f il e sy s tem on kafka-0000 i snearly  full ! (87%)

Oh,shit. I me a n ,  uhm, wh at a gr e at opportunity !

Th at’s quite an odd e rrorto see!Kafka isa distribu ted  lo g.AtK e en , we write in c o ming  e vents t o K afka as  asor t o f h ol ding tank  until w eca nwr ite  them into Cas sa ndr afor  perm a nent s torag e. K af ka isconfigu r ed t o re ta in  a r oll in g 168  hou r s —  as som e peoplecal l  it,  a  w eek — o f data just i ncase we ne ed  to  refer b ack to i t. The significance oft his err orwas t hatwe no lon ger  had e nou g h spac e to a ccomm od ate a  week o fcus to mer data!H o w d id th is  hap pen?

T he Source of Oppor t uni ty

After rea chi ng o u t  t o s o me tea mm a tes , I learned thatone of our  c u st omers hadr ele as ed a ne w mobil e gam e  over t he weeken d that be en  a g r e atsuccess. A wesome!The downs i de wa s  t hat t hehuge w av e o f inb ound d a tafromthat g am e wasexcee di ng our s hort-term storage.

Weneeded  to g etthis work edo utfast, but  our  options w ere  li mited . S e e, th e ve r si ono f Kaf ka w e  were usi n gatt h etime — 0.7. x —  di dn’tall o w youto c hange retention c o nfiguration wi thou t res tar t ing.  W e couldn’t doa r es t a rt due t oa lack of r eplicati on ;  agai n a Ka fka 0. 7.x thi n g .I f thedisk s f illed upwit ho u t a fix ,we 
re trying_to run_a bunch of computers_toward a_common_goal. He_was_fond of saying:_“opportunities arise”.

This summer_at Keen IO, we_a had a_series_of postmortem-worthy “opportunities” that threatened everything we hold near and dear: the stability of_our_platform, the_trust_of_our customers, and the sanity_of our team. Please allow_us to_tell you about them in cathartic detail.

Opportunity Knocks_and_It’s a Closet_And It’s Full…

The morning of July 21st, I was_working from home, because I had_a flight to_O’Reilly’s_OSCON_later that day. With_some time to kill, I was_looking over various charts and alerts_for our infrastructure, when I noticed there_was a muted alert that had_been triggered. We generally mute_alerts when_we don’t want to be_bugged with an_email or_a phone call_temporarily. Unfortunately, this alert had been_forgotten, and it_just so happened to be one_of_the most dreaded_in_all_of computerdom:

The_filesystem on kafka-0000_is_nearly full!_(87%)

Oh,_shit. I mean, uhm, what a_great_opportunity!

That’s quite an odd error to see!_Kafka is a distributed_log._At Keen, we write_incoming events to Kafka as_a sort of holding tank until_we can_write them_into Cassandra for permanent storage. Kafka is configured to retain a_rolling 168 hours — as some_people call it, a_week —_of_data just in_case_we need_to refer back to it. The significance_of this_error was that we no longer_had enough space to_accommodate_a week of customer data! How_did this happen?

The Source of Opportunity

After_reaching out to some teammates,_I_learned_that one of our customers_had released a new mobile game_over the weekend_that been a great success. Awesome! The_downside_was that the huge wave of_inbound_data from that game was exceeding_our_short-term_storage.

We needed to get this_worked out fast, but our options_were limited. See, the version of Kafka we were_using at the_time — 0.7.x — didn’t_allow_you_to change retention configuration without restarting. We couldn’t do a_restart due_to a lack_of replication; again a Kafka 0.7.x thing. If the disks_filled up without a fix, we