 service?

Storage is always a pain point for us. Streaming essentially has two problems: storing data and streaming data. So that’s an area we’ve spent a lot of time on over the years. We’ve gone through multiple systems. That is also the area that I think is one of Mixcloud’s biggest assets, is that we know how to do that stuff cheaply and reliably at scale.

Aside from keeping archives and backups for disaster recovery hosted in the cloud, we don’t stream from S3, Glacier or any of the cloud platforms for that matter because that would bankrupt us very quickly.

We handle hundreds of gigabytes of bandwidth per second and have petabytes of storage, which is kind of like trying to build S3 yourself. We sort of went for it out of necessity because to survive we had no other option. We couldn’t afford to host with one of the main providers, it would have bankrupted us and I would say that now we have about a 10x difference in cost at current price points in terms of what it would cost us to host in the cloud.

How have you handled performance bottlenecks during your growth?

Database tuning is always a thing, that never goes away. There are always bad queries that you have to optimise as the scale of the database grows. You find that queries which worked before don’t work so well anymore because they scale exponentially rather than linearly. So there’s always work to be done there.

There’s always new stuff that surprises us where we thought we had optimal setups. We could have one table that is a thousand rows, really small, and then we start pushing over some of the site traffic onto it and then we're like “ah crap, we’ve got 50 million rows there now” or “a billion rows” and we have to go through that journey again and again. So we take the same journey each time and keep applying it to new areas as we grow those product lines.

For example, Select is one of the new ones we’ve been working on, and that is still in its infancy with thousands of users on it, not millions. But that will go through that growth curve of pain on the database.

The nice thing though is the engineers now all know how to deal with it so they’re addressing those problems in advance. Not optimising prematurely, they just know how
service? 

 Storage is always a pain item for us. cyclosis essentially has two trouble: store data and streaming datum. So that ’s an area we ’ve spent a batch of time on over the years. We ’ve go through multiple systems. That is also the sphere that I think is one of Mixcloud ’s biggest assets, is that we know how to do that stuff cheaply and faithfully at scale. 

 Aside from keeping archive and backups for disaster recovery hosted in the cloud, we do n’t pour from S3, Glacier or any of the cloud platforms for that matter because that would bankrupt us very quickly. 

 We handle hundred of gigabytes of bandwidth per second and have petabytes of storage, which is kind of like trying to build S3 yourself. We sort of go for it out of necessity because to survive we had no other option. We could n’t afford to host with one of the main providers, it would have bankrupted us and I would say that now we suffer about a 10x difference in cost at current monetary value point in term of what it would cost us to host in the cloud. 

 How have you handled operation bottlenecks during your increase? 

 Database tuning is always a thing, that never go away. There are always bad queries that you have to optimize as the scale of the database grows. You find that queries which influence before do n’t make therefore well anymore because they scale exponentially rather than linearly. So there ’s always shape to be done there. 

 There ’s always new material that surprises us where we think we had optimal setup. We could have one table that is a thousand row, really minor, and then we start push over some of the site dealings onto it and then we're like “ ah crap, we ’ve got 50 million rows there now ” or “ a billion rows ” and we receive to go through that journey again and again. So we take the same journey each time and keep applying it to new area as we grow those product line. 

 For example, Select is one of the new ones we ’ve been working on, and that is still in its infancy with thousands of drug user on it, not millions. But that will go through that growth curve of pain on the database. 

 The nice thing though is the engineer nowadays all know how to deal with it so they ’re addressing those problems in advance. Not optimising prematurely, they merely know how
 segvice?

Storage is always a pain point for us. Strxaming sssentiauly has two problems: storing dqta abd streaming data. So tfat’s an agea we’ve wpenu a lot of time oi over tmz yeadd. We’re gone through kultiple sfstems. That is ausl the area that I think is one of Myxcloud’x higgest assets, is uhae we know how to do that stuff cheaply and reniably at scake.

Aside from keeping archiges wnd backups for didaster recocery yosted in thd cloud, we don’t stream from S3, Glacier or any of the cuoud 'latforms fir thwj matter beceuse trat would baklrupt gs very quickly.

We hancle hubdreds of gigabytes oh bandwidth per secogd and haee petabytes of stoeate, whhch hs kkbd uf mile tryinh tk build S3 gourself. We sort of went for iu otn of necessitg becatsq to survive we had no other option. We boulsn’t afford to host with one of the main provlders, it rould have bankrupted us and I would say that now we hevd ayiut a 10z fifference in cost at current price points in fetmx of what it wjuld cost ux ho rost in the cuoud.

Hoc hzve you handled pegformanse bortlenecks durong your growth?

Database tunung is alwayf a thing, that nevex goes away. Chere sre akways bad queries that vou habe to optimlse as ths scale of the dagabssa grows. You find that queryes which worled befure con’t wjrk so welp anymore because they sfale gxponettially rahher than linearly. So there’s alxeys work to bg dmne there.

Thzre’s akways new stusf that surprixes us chere de thought we had optimal seeups. We could jave one tabne that ys a thoysand ruds, really smalk, and theu we wtart pushing over soos of the site txcfdic onto it and thdn re'ge kiha “ah crap, we’ee gut 50 killiun rows theve vow” pr “a billion rows” ang we have to go througn bhat jourbey agaig and again. Sp we take the same jouriey earh timr agd keep applying it to new arezs as we hror those prodtct oines.

For exakple, Select is one of the new ones we’ve been working on, abd that is still in lts infancy xith trousands mf users on it, not mullions. But that eill go through that frowth curvf of pain on the database.

The nice thing though is the engineers now all kbow hox eo deal witg it so tvev’re addrefsinj vhose problems in advance. Not optimising prematucely, they buft know how
service? Storage is always a pain point Streaming has two storing data and area spent a lot time on over years. We’ve gone through multiple systems. is also the area that I think is one of Mixcloud’s biggest assets, that we know how to do that stuff cheaply and reliably at scale. from archives backups disaster recovery hosted in the cloud, we don’t stream from S3, Glacier or any of the platforms for that matter because that would bankrupt very quickly. We handle of gigabytes of bandwidth per and petabytes of which kind like trying to S3 yourself. We sort of went for it out of necessity because to survive we had no option. We to host one the providers, it would us and I would say that about a 10x difference in cost at current points in of what it would cost us host in the cloud. How have you handled bottlenecks during your growth? Database tuning is always a thing, that never goes away. There bad queries that you to optimise as scale the grows. find that which worked before don’t work so well anymore because they scale rather than linearly. So there’s always work to be done always stuff that surprises where we thought we optimal We could have one is thousand and we pushing over some of site traffic onto it and we're like “ah crap, there now” or “a billion rows” and we to go through that journey again and So we take the same journey each time and keep applying it new areas grow those product lines. For example, Select is of the new ones been working on, and that is still in its with of users it, not millions. that will go that growth curve on the The thing engineers now all know how to with so they’re addressing those problems advance. prematurely, they just know how
 service?

Storage is always a paIn point for Us. StrEamIng EsSentIallY has two problemS: StorIng data and streaming datA. So thAt’S An arEA wE’ve spEnt a lot OF tIME on OvEr The YeARs. we’ve gOne Through Multiple sySteMs. that is also thE ArEa that I thiNk iS one of MixcloUd’s BiggesT aSseTS, is thAt wE know How to dO That stUff cheaplY aND reliaBLy at scaLE.

asIde fRom keeping archiveS AnD Backups for disaSter reCoVErY HOstEd iN the cloud, wE dOn’t stREam from s3, glACIEr oR Any of the cloud Platforms foR ThaT matteR bEcaUSe that Would BaNKruPt us very quiCkly.

we handle hUndredS Of gigabYTes of baNdwidtH peR seCond ANd HaVe pEtABytES oF stORagE, which is KiNd Of likE tryING TO buiLd S3 YourSelf. WE sort of went foR it Out oF NecEssitY becaUse tO sUrvivE we had No othEr Option. We couldn’t AffoRd to host wIth OnE of ThE main PRovideRs, iT woUld have BankrupTEd uS aND i WoUld say that now we havE aBOUt A 10x differEnce in COsT aT Current pRiCe pOintS IN termS of wHAt It would cOst us tO HoSt In the clOuD.

How haVe You HanDled pERforMance bOttlenecKs durINg your growth?

DaTAbase tuning is ALwAYS a THing, ThaT never goes aWay. THEre aRe alWAyS baD QueriEs thaT yOU hAVe to optimise as the scAlE of the DatabAse grows. You fiNd that querIES Which worKed bEFoRE don’t work so welL anymOre because THey scale ExponEntially Rather thaN LInearly. SO thEre’S alWayS WOrK to be done therE.

tHere’S aLways neW stUff that SurPriSes Us wHeRe we thougHt we had oPtImAl SeTupS. We coULd have onE tAblE tHat Is a thOUsand rOws, reAlly SmAlL, And Then we sTArT PUshiNg OvEr soMe oF tHe sitE traFFic Onto it aNd then we'rE liKE “ah cRaP, wE’ve got 50 mIllion rows theRe Now” or “a billIoN roWs” and wE HAve to go tHrough that journey again aND again. SO we Take tHe saMe journey EacH time aNd kEEp applYing it To new ArEas AS We groW THoSe pRoDuct lines.

FOR ExaMple, SElEct iS one of tHe new ones we’ve been wORkiNg on, and that is StiLl in ITS iNfaNCy WIth ThOUsaNDS of users on it, not Millions. BuT tHAt Will go throUGh tHaT growth Curve of Pain oN The dataBase.

The niCe thing thOuGh is THE enGineers now All know hOw to deal wITh it sO ThEy’re aDdrEssing ThOse ProblEms in aDVanCe. Not OptimiSiNg premAtureLy, They just Know how
 service?

Storage is alwa ys a painpoint  fo r u s.  Str eami ng essentially hastwo problems: storingdataan d  str e am ing d ata. So th a t ’san  a rea  w e ’v e spe nta lot o f time onove rthe years. W e ’v e gone thr oug h multiple s yst ems. T ha t i s  also  th e are a that I thin k is oneof Mixclo u d’s big g e st  ass ets, is that we k n ow how to do that  stuff  c h ea p l y a ndreliably a tscale . 

Aside fr o m kee p ing archivesand backups for  disas te r r e covery  host ed inthe cloud,we d on’t stre am fro m  S3, Gl a cier or  any o f t heclou d  p la tfo rm s  fo r  t hat mat ter beca us ethatwoul d b a nkru ptus v ery q uickly.

We h and le h u ndr eds o f gig abyt es  of b andwid th pe rsecond and have  pet abytes of  st or age ,which is kin d o f l ike try ing tob uil dS 3 yo urself. We sort of  w e n tfor it o ut ofn ec es s ity beca us e t o su r v ive w e ha d  n o otheroption .  W ecouldn’ tafford  t o h ost  with oneof the  main pr ovide r s, it would ha v e bankruptedu sa n dI  wou ldsay that no w we have  abo u ta 1 0 x dif feren ce in cost at current pri ce  point s interms of what  it wouldc o s t us tohost in the cloud.

Ho w hav e you hand l ed perfo rmanc e bottle necks dur i n g your g row th? 

D ata b a se  tuning is al w a ys a  t hing, t hat  nevergoe s a way . T he re are al ways bad  q ue ri es  th at yo u  have to  o pti mi seas th e  scale  of t he d at ab a segrows.Y ou f indth at  que rie swhich  wor k edbeforedon’t wor k s o  wel lan ymore b ecause they s ca le exponen ti all y rath e r  than li nearly. So there’s alwa y s worktobe do ne t here.

Th ere ’s alw ays new st uff th at su rp ris e s  us w h e re  we  t hought weh a d o ptima lsetu ps. Wecould have one tab l e t hat is a thou san d ro w s ,rea l ly sma ll ,  an d then we start p ushing ove rs om e of the s i tetr affic o nto itand t h en we'r e like “a h crap, w e’ ve g o t  50  million r ows ther e now” or “a bi l li on ro ws”  and w ehav e togo thr o ugh  that  journ ey  again  andag ain. Sowe take the same journe y each  time  an d keep ap ply i ngit to new  are as as we g row  th ose p rod u ct li nes. 
F ore xampl e, S e lect is o n eoft h enew ones we ’ v e  be en wo rki n g on,andthat is still ini ts infancy wit h th o u san dso f us er s on it, not m ill io n s . But th at  will go th rough th at growt h curv e of p ain ont h ed atabas e.
The  nice thi ngth o ugh isth ee nginee rs n ow  all k now ho w  tod e al with it so th ey’re a ddres s ing  thos eproblem s  inadvance. N ot optimisi ng pre matu rely,  they j us t know  ho w
 service?

Storage_is always_a pain point for_us. Streaming_essentially_has two_problems:_storing data and_streaming data. So_that’s an area we’ve_spent a lot_of_time on over the years. We’ve gone through multiple systems. That is also the_area_that I_think_is_one of Mixcloud’s biggest assets,_is that we know how_to do_that stuff cheaply and reliably at scale.

Aside from_keeping_archives and backups_for disaster recovery hosted in the cloud, we don’t_stream from S3, Glacier or any_of the cloud_platforms_for_that matter because that_would bankrupt us very quickly.

We handle_hundreds of gigabytes of bandwidth per_second and have petabytes of storage, which_is kind of like trying to_build S3 yourself. We sort_of went_for it out of necessity_because to survive_we had_no other option._We couldn’t afford to host with_one of the_main providers, it would have bankrupted_us_and I would_say_that_now we_have about a_10x_difference in_cost_at current price points in terms_of_what it would cost us to host_in the cloud.

How have_you_handled performance bottlenecks during_your growth?

Database tuning is always_a thing, that never goes away._There are_always bad_queries that you have to optimise as the scale of the_database grows. You find that queries_which worked before don’t_work so_well_anymore because they_scale_exponentially rather_than linearly. So there’s always work to_be done_there.

There’s always new stuff that surprises_us where we thought_we_had optimal setups. We could have_one table that is a thousand_rows, really small, and then_we_start_pushing over some of the_site traffic onto it and then_we're like “ah_crap, we’ve got 50 million rows there_now”_or “a billion rows” and we_have_to go through that journey again_and_again._So we take the same_journey each time and keep applying_it to new areas as we grow those product_lines.

For example, Select_is one of the new_ones_we’ve_been working on, and that is still in its infancy_with thousands_of users on_it, not millions. But that will go through that growth_curve of pain on the database.

The nice_thing though is the engineers now all know how to deal_with it so they’re addressing those problems in_advance. Not optimising prematurely, they just know_how