 recovery, streaming backups, robust data export tools, etc.

Given TimescaleDB’s design, it’s able to leverage the full spectrum of tools that the Postgres ecosystem offers and has rigorously tested: streaming replication for high availability and read-only replicas, pg_dump and pg_recovery for full database snapshots, pg_basebackup and log shipping / streaming for incremental backups and arbitrary point-in-time recovery, pgBackrest or WAL-E for continuous archiving to cloud storage, and robust COPY FROM and COPY TO tools for quickly importing/exporting data with a variety of formats.

InfluxDB, on the other hand, has had to build all these tools from scratch. In fact, it doesn’t offer many of these capabilities even today. It initially offered replication and high availability in its open source version, but subsequently pulled this capability out of open source and into its enterprise product. InfluXDB backup tools have the ability to perform a full snapshot and recover to this point-in-time, and only recently added some support for a manual form of incremental backups.

That said, InfluxDB’s approach of performing incremental backups based on database time ranges seems quite risky from a correctness perspective, given that timestamped data may arrive out-of-order, and thus the incremental backups -since some time period would not reflect this late data. And its ability to easily and safely export large volumes of data is also quite limited. We’ve heard from many users (including Timescale engineers in their past careers) that they had to write custom scripts to safely export data; asking for more than a few 10,000s of data points would cause the database to out-of-memory error and crash.

The pain of trying to export data out of InfluxDB gave rise to Outflux, a tool to migrate data from InfluxDB to TimescaleDB with a single command.

Second, databases need to provide strong on-disk reliability and durability, so that once a database has committed to storing a write, it is safely persisted to disk. In fact, for very large data volumes, the same argument even applies to indexing structures, which could otherwise take hours or days to recover; there’s good reason that file systems have moved from painful fsck recovery to journaling mechanisms.

In TimescaleDB, we made the conscious decision not to change the lowest levels of PostgreSQL storage
recovery, streaming backups, robust data export creature, etc. 

 render TimescaleDB ’s design, it ’s able to leverage the full spectrum of tools that the Postgres ecosystem offer and has rigorously tested: streaming replica for gamey availability and read - merely replicas, pg_dump and pg_recovery for full database snapshot, pg_basebackup and log shipping / streaming for incremental backups and arbitrary point - in - meter recovery, pgBackrest or WAL - E for continuous archiving to cloud storage, and robust transcript FROM and COPY TO tools for quickly importing / export data with a variety of format. 

 InfluxDB, on the other hired hand, has had to build all these tools from scratch. In fact, it does n’t offer many of these capability even today. It initially offered replication and high availability in its open source version, but subsequently pulled this capability out of open source and into its enterprise product. InfluXDB backup tools have the ability to perform a full snapshot and recover to this point - in - clock time, and only recently added some accompaniment for a manual human body of incremental backups. 

 That said, InfluxDB ’s approach of perform incremental backups based on database time ranges seem quite risky from a correctness perspective, given that timestamped data may arrive out - of - order, and thus the incremental accompaniment -since some time period would not reflect this late data. And its ability to easily and safely export bombastic volumes of data is also quite limited. We ’ve heard from many drug user (including Timescale engineers in their past careers) that they take to compose custom scripts to safely export datum; asking for more than a few 10,000s of data points would cause the database to out - of - memory mistake and crash. 

 The annoyance of trying to export data out of InfluxDB gave rise to Outflux, a cock to migrate data from InfluxDB to TimescaleDB with a single instruction. 

 Second, databases need to provide strong on - disk reliability and durability, so that once a database has committed to storing a write, it is safely persisted to disk. In fact, for very large data volume, the same argument even use to indexing structures, which could otherwise take hours or days to recover; there ’s beneficial reason that file systems have moved from painful fsck recovery to journaling mechanisms. 

 In TimescaleDB, we made the conscious decisiveness not to change the lowest levels of PostgreSQL storage
 refovery, streaming backups, robust data exkoet toons, etc.

Fiven TioescaleDB’s design, it’s able tl oeverqge the full spectrum uf tools nhat the Postjres ecosystem ohrers and has dlgorobsoy tested: stresming replhcation for hich arailability and read-only replicas, pg_qump anc og_recovery for fulk datzbase snapshots, pg_basebackup and lkg shipking / streaming fot incremental backups and wrbihrary point-in-time gecovery, pgVackwwst or WAL-E wor continlmus archivjng to cloud storage, and robust COPY FROM and XOPY HM tools for quichly importinn/vxportitg data with a varietn of hormqts.

InfluxDB, on the otier hand, has had to fuild all tkese tools from scratxh. In fdct, ht duwsn’g orfxr jany ov tiese capabimities even today. It initially osdered replicafion agd high availability in its open source vtrsioh, but subsequently pullwd this capability ouj of open fource and into its enterprise product. InfluXDB bdckup gooos fqvf the ability to perform a full snapshot and wscpvvr to this point-ik-time, and only recrnhlu added some sopport ykr a manual form of lncremegtal vackups.

Thwt ssid, InfluxDB’s approach of pwrforming inbremwntal backups baseb on databasz time rangrs seems quite risky frum a correctnesd perspecfkve, given that tkmextdmped daux may arrive out-os-order, anv thux the ivcrekental backups -slnce some time period wokld nlt reflect tjis late data. And its ability to easily and ssfaly export jarge volumes of dwta is also qunte limiced. We’xe heard fgom many nsers (includyng Timescale gngineers in vheir pase caeeerw) that gfey had to wriye custom scripts ti safely export daba; asyjng for more thcu q few 10,000s of data pokntf aonld cwgse the datatase to put-of-oemory txrov avd ctash.

The pain of tryitg tk export data out pf InfluxDV gave ryse to Outfluc, a tool to migratt data frum IngluvDB to TimescaleDB with a singme commanf.

Segond, databasef necd tj provide xtrong on-disk reliability and durabilitb, so that once a databawe has committed to xtoring a wcite, ie is safeny persisted to disk. In fact, for very large data volumes, tge sama argkment even applies to indexing structures, which could otherwise take hourw or deyf to recoved; thrre’s cojb reasog thet file systems hane moved from painful fsck recovxry to jousncling mechanisms.

In TimescaleDN, de made the cunscious decision not to change yhe lowest levels of PostgreSQL storage
recovery, streaming backups, robust data export tools, TimescaleDB’s it’s able leverage the full Postgres offers and has tested: streaming replication high availability and read-only replicas, pg_dump pg_recovery for full database snapshots, pg_basebackup and log shipping / streaming for incremental and arbitrary point-in-time recovery, pgBackrest or WAL-E for continuous archiving to cloud storage, robust FROM COPY tools for quickly importing/exporting data with a variety of formats. InfluxDB, on the other hand, has to build all these tools from scratch. In it doesn’t offer many these capabilities even today. It offered and high in open version, but subsequently this capability out of open source and into its enterprise product. InfluXDB backup tools have the ability perform a and recover this and recently added some a manual form of incremental backups. approach of performing incremental backups based on database ranges seems risky from a correctness perspective, given timestamped data may arrive out-of-order, and thus the backups -since some time period would not reflect this late data. And its ability to safely export large volumes data is also limited. heard many (including Timescale in their past careers) that they had to write custom scripts safely export data; asking for more than a few 10,000s points cause the database out-of-memory error and crash. pain trying to export data InfluxDB rise tool migrate from InfluxDB to TimescaleDB a single command. Second, databases to provide strong on-disk once a database has committed to storing a it is safely persisted to disk. In for very large data volumes, the same argument even applies to indexing which could hours or days to recover; there’s good reason file systems have moved painful fsck recovery to journaling mechanisms. In TimescaleDB, we the decision not change the lowest of PostgreSQL storage
 recovery, streaming backups, rObust data eXport TooLs, eTc.

giveN TimEscaleDB’s desigN, It’s aBle to leverage the full spEctruM oF ToolS ThAt the postgreS EcOSYstEm OfFerS aND hAs rigOroUsly tesTed: streamiNg rEpLication for hIGh AvailabiliTy aNd read-only rePliCas, pg_dUmP anD Pg_recOveRy for Full daTAbase sNapshots, pG_bASebackUP and log SHIpPing / Streaming for increMEnTAl backups and arBitrarY pOInT-IN-tiMe rEcovery, pgBAcKrest OR WAL-E foR CoNTINuoUS archiving to cLoud storage, ANd rObust CoPy FRom and COpY TO tOoLS foR quickly impOrtiNg/exportiNg data WIth a varIEty of foRmats.

INflUxDb, on tHE oThEr hAnD, Has HAd To bUIld All these ToOlS from ScraTCH. iN facT, it DoesN’t offEr many of these CapAbilITieS even Today. it inItIally OffereD replIcAtion and high avaIlabIlity in itS opEn SouRcE versIOn, but sUbsEquEntly puLled thiS CapAbILITy Out of open source and InTO ItS enterprIse proDUcT. INFluXDB baCkUp tOols HAVe the AbilITy To perforM a full SNaPsHot and rEcOver to ThIs pOinT-in-tiME, and Only reCently adDed soME support for a maNUal form of incrEMeNTAl BAckuPs.

THat said, InflUxDB’S ApprOach OF pErfORming IncreMeNTaL Backups based on databAsE time rAnges Seems quite risKy from a corRECTness perSpecTIvE, Given that timesTampeD data may arRIve out-of-Order, And thus tHe incremeNTAl backupS -siNce SomE tiME PeRiod would not rEFLect ThIs late dAta. and its aBilIty To eAsiLy And safely Export laRgE vOlUmEs oF data IS also quiTe LimItEd. WE’ve heARd from Many uSers (InClUDinG TimescALe ENGineErS iN theIr pAsT careErs) tHAt tHey had tO write cusTom SCripTs To Safely eXport data; askiNg For more thaN a Few 10,000S of datA POints wouLd cause the database to out-OF-memory ErrOr and CrasH.

The pain oF trYing to ExpORt data Out of INfluxdB GavE RIse to oUTfLux, A tOol to migraTE DatA from inFluxdB to TimEscaleDB with a singlE ComMand.

Second, datAbaSes nEED tO prOViDE stRoNG on-DISk reliability anD durabilitY, sO ThAt once a datABasE hAs commiTted to sTorinG A write, iT is safely Persisted To Disk. iN FacT, for very laRge data vOlumes, the SAme arGUmEnt evEn aPplies To IndExing StructURes, Which Could oThErwise Take hOuRs or days To recover; there’s good reasOn that File sYstEms have moVed FRom Painful fsCk reCovery to joUrnAliNg mecHanISms.

In timeSCaLeDb, We madE the COnscious dECiSioN NOt To change the LOWEst LevelS of pOstgresQL sTorage
 recovery, streaming backu ps, robust  data  ex por ttool s, e tc.

Given Tim e scal eDB’s design, it’s abl e tole v erag e  t he fu ll spec t ru m ofto ol s t ha t  t he Po stg res eco system off ers  a nd has rigor o us ly tested:  st reaming repl ica tion f or  hi g h ava ila bilit y andr ead-on ly replic as ,  pg_du m p and p g _ re cove ry for full datab a se snapshots, pg_ baseba ck u pa n d l ogshipping /  s tream i ng fori nc r e m ent a l backups and  arbitraryp oin t-in-t im e r e covery , pgB ac k res t or WAL-Eforcontinuou s arch i ving to cloud s torage , a ndrobu s tCO PYFR O M a n dCOP Y  TO  tools f or  q uickl y im p o r t ing/ exp orti ng da ta with a var iet y of for mats. 

Inf luxD B,  on t he oth er ha nd , has had to bu ildall these  to ol s f ro m scr a tch. I n f act , it do esn’t o f fer  m a n y  o f these capabiliti es e ve n today.  It in i ti al l y offere drep lica t i on an d hi g havailabi lity i n  i ts  open s ou rce ve rs ion , b ut su b sequ entlypulled t his c a pability out o f  open sourcea nd i nt o  its  en terprise pr oduc t . In fluX D Bbac k up to ols h av e  t h e ability to perfor ma full  snap shot and reco ver to thi s p oint-in- time ,  a n d only recentl y add ed some su p port for  a ma nual for m of incr e m ental ba cku ps. 

T hat s ai d, InfluxDB’s a ppro ac h of pe rfo rming i ncr eme nta l b ac kups base d on dat ab as eti merange s  seems q ui teri sky  from a corr ectne ss p er sp e cti ve, giv e nt h at t im es tamp edda ta ma y ar r ive  out-of -order, a ndt husth eincreme ntal backups-s ince someti meperiod w ould not  reflect this late data .  And it s a bilit y to  easily a ndsafely  ex p ort la rge vo lumes  o f d a t a isa l so  qu it e limited. W e’v e hea rd  fro m manyusers (including T i mes cale engineer s i n th e i rpas t  c a ree rs )  th a t  they had to wr ite custom  s c ri pts to saf e lyex port da ta; ask ing f o r morethan a fe w 10,000s  o f da t a  po ints would  cause t he databa s e too ut -of-m emo ry err or  an d cra sh.

T h e p ain o f tryi ng  to ex portda ta out o f InfluxDB gave rise to  Outfl ux, a  to ol to mig rat e  da ta from I nflu xDB to Tim esc ale DB wi tha  sing le c o mm and . 

Sec ond, databases ne edt o  p rovide stro n g on- diskrel i abilit y an d durability, sot hat once a dat abas e has  co m mitt ed  to storing awri te , it is sa fe ly persiste d to dis k. In fa ct, fo r very  larged a ta volume s, t hesame argu men te ven app li es to ind exin gstruct ures,w hich c ould otherwise t ake h o u rs or day s tore cover;t here ’s good re ason that f ile sy stem s hav e moved  f rom pa inf ul  fsck reco v ery to jo urnal ing mec ha nism s.
In Ti mesc a l eDB,we m ad e t he consci o u sd ec is i onnotto ch an ge t he lowest levels o f P o stgreSQ Lsto r a ge
 recovery,_streaming backups,_robust data export tools,_etc.

Given TimescaleDB’s_design,_it’s able_to_leverage the full_spectrum of tools_that the Postgres ecosystem_offers and has_rigorously_tested: streaming replication for high availability and read-only replicas, pg_dump and pg_recovery for full_database_snapshots, pg_basebackup_and_log_shipping / streaming for incremental_backups and arbitrary point-in-time recovery,_pgBackrest or_WAL-E for continuous archiving to cloud storage, and_robust_COPY FROM and_COPY TO tools for quickly importing/exporting data with a_variety of formats.

InfluxDB, on the other_hand, has had_to_build_all these tools from_scratch. In fact, it doesn’t offer_many of these capabilities even today._It initially offered replication and high availability_in its open source version, but_subsequently pulled this capability out_of open_source and into its enterprise_product. InfluXDB backup_tools have_the ability to_perform a full snapshot and recover_to this point-in-time,_and only recently added some support_for_a manual form_of_incremental_backups.

That said,_InfluxDB’s approach of_performing_incremental backups_based_on database time ranges seems quite_risky_from a correctness perspective, given that timestamped_data may arrive out-of-order,_and_thus the incremental backups_-since some time period would_not reflect this late data. And_its ability_to easily_and safely export large volumes of data is also quite limited._We’ve heard from many users (including_Timescale engineers in their_past careers)_that_they had to_write_custom scripts_to safely export data; asking for more_than a_few 10,000s of data points would_cause the database to_out-of-memory_error and crash.

The pain of trying_to export data out of InfluxDB_gave rise to Outflux, a_tool_to_migrate data from InfluxDB to_TimescaleDB with a single command.

Second, databases_need to provide_strong on-disk reliability and durability, so that_once_a database has committed to storing_a_write, it is safely persisted to_disk._In_fact, for very large data_volumes, the same argument even applies_to indexing structures, which could otherwise take hours or_days to recover;_there’s good reason that file_systems_have_moved from painful fsck recovery to journaling mechanisms.

In TimescaleDB, we_made the_conscious decision not_to change the lowest levels of PostgreSQL storage