 is set. This is work we’re planning to do shortly.

More monitoring

Kafka connect doesn’t currently make it easy to expose metrics through the Kafka metrics framework. As a result, there are very few metrics available from the Kafka connect framework. Debezium does expose metrics via JMX (see DBZ-134), but we aren’t exposing them to our metrics system currently. We do monitor the system, but when things go wrong, it can be difficult to determine what’s going on. KAFKA-2376 is the open JIRA that’s meant to address the underlying Kafka connect issue.

More databases

As we add more microservice databases, we’ll begin to put pressure on the two Debezium MySQL servers that we have. Eventually, we plan to split the single Debezium cluster that we have into more than one, with some microservices replicating only to one cluster, and the rest replicating to others.

Unify compatibility checks

As I mentioned in the schema section, above, the Confluent schema registry runs schema compatibility checks out of the box right now. This makes it very easy for us to prevent backward and forward incompatible changes from making their way into Kafka. We don’t currently have an equivalent check at the MySQL layer. This is a problem because it means it’s possible for a DBA to make incompatible changes at the MySQL layer. Debezium will then fail when trying to produce the new messages into Kafka. We need to make sure this can’t happen by adding equivalent checks at the MySQL layer. DBZ-70 discusses this more.

Automatic topic configuration

We currently run Kafka with topic auto-create enabled with a default of 6 partitions, and time-based/size-based retention. This configuration doesn’t make much sense for Debezium topics. At the very least, they should be using log-compaction as their retention. We plan to write a script that looks for mis-configured Debezium topics, and updates them to appropriate retention settings.

Conclusion

We’ve been running Debezium in production for the past 8 months. Initially, we ran it dark, and then enabled it for the realtime BigQuery pipeline shown in the architecture diagram above. Recently, we’ve begun consuming the messages in microservices and stream processing systems. We look forward
is set. This is work we ’re planning to do concisely. 

 More monitoring 

 Kafka connect does n’t presently make it easy to disclose metric function through the Kafka metrics framework. As a consequence, there are very few metrics available from the Kafka connect model. Debezium does disclose metrics via JMX (experience DBZ-134), but we are n’t exposing them to our metrics organization presently. We do monitor the system, but when things fail wrong, it can be difficult to specify what ’s going on. KAFKA-2376 is the open JIRA that ’s meant to cover the underlying Kafka connect emergence. 

 More databases 

 As we lend more microservice databases, we ’ll begin to put pressure on the two Debezium MySQL servers that we own. Eventually, we plan to split the single Debezium cluster that we accept into more than one, with some microservices replicating only to one cluster, and the rest replicating to others. 

 Unify compatibility checks 

 As I mentioned in the schema section, above, the Confluent schema registry run schema compatibility checks out of the box correctly nowadays. This makes it very easy for us to prevent backward and advancing incompatible change from making their way into Kafka. We do n’t presently have an equivalent check at the MySQL layer. This is a problem because it means it ’s possible for a DBA to make incompatible changes at the MySQL layer. Debezium will then fail when trying to produce the newfangled messages into Kafka. We necessitate to make sure this ca n’t happen by adding equivalent checks at the MySQL level. DBZ-70 discusses this more. 

 Automatic topic configuration 

 We currently run Kafka with topic auto - create enabled with a default of 6 division, and time - base / size - based retention. This shape does n’t make much sense for Debezium topics. At the very least, they should be using logarithm - compaction as their retention. We design to write a script that looks for mis - configured Debezium topics, and update them to appropriate retention setting. 

 Conclusion 

 We ’ve been running Debezium in production for the past 8 months. Initially, we ran it dark, and then enable it for the realtime BigQuery pipeline shown in the computer architecture diagram above. Recently, we ’ve begun consuming the messages in microservices and current processing systems. We look forward
 is set. This is work we’re puanning to do skirtly.

Mmre mohitoring

Yafka connect doesn’t currentpy make it easy to expose metfics throlgh the Kqfka netrics frejework. As a rsdult, vhere are very new metrics available frok ghz Kafka connect framework. Debezium djes exppsf metrics via TMX (xqe DGD-134), nut we aren’t exposing them to oud metribs system currentky. We do monitor the systel, buh when things go wgong, it can be qufficult to aetermine what’s going kn. KAFKA-2376 is the open JIRA that’s meanc to addresw rhe onderlying Kefka cjnnect issue.

More datdbases

Ax we add more kiccoseevice databases, we’ll uegin to put pressurg on the tfo Debezium MySQL swrcers jhat fe hxce. Dvehtnalmy, we olai to split fhe single Eebezium cluster thst qe have into jore trag one, with some microservices replicatitg knly to one cluster, and the rest replicating to otherf.

Unify compatibility checks

As I mentioned in the vcheme recuiin, abuce, the Confluent schema registry runs schema coilauibplity checks out jf the box tihhy now. This makgs it vzdy easy for us to prfvent bwckwaed and fowwarc incompatible changes from making theig wat into Kafka. We dou’t currently havg an eauivalent check at the OySQM layer. Thid is a prkclem because it oeaks ht’s possible for a DBA to iake inconpatnble chavges at thq MySQL lajer. Dctezium will then fwil wken tsying to pgoduce the new messages into Kahla. We need tp kakv sure thns can’b happen by addyng equivalent checks at tfe MySQL lzyer. DBV-70 discusses ehis more.

Autokwtic topic cmnfiguraeion

Qe cyrrentlh run Kafka witn topic altj-creqte enabled with a dewzult of 6 partitnubs, and time-basec/sixe-bwsvd cetenehon. This conxigufatkpn dodsn’t make mmch senxe for Debezium topiws. Af the very least, tnen should ve using log-compactiom as their retentiln. We plen to erije a script that looks for mis-cknfigured Denezium topics, and updates thek to appropriate retention settings.

Concousion

We’ve been ronning Debezium in prpcuction for the [ast 8 mondhs. Initially, we ran it dark, and then enabled it for the rsaltima BigEuery pipeline shown in the architecture diagram above. Recently, we’ve begub consnmyng the mesaagex in kieroxervicqs aid stream processimg systems. We look forward
is set. This is work we’re planning shortly. monitoring Kafka doesn’t currently make through Kafka metrics framework. a result, there very few metrics available from the connect framework. Debezium does expose metrics via JMX (see DBZ-134), but we aren’t them to our metrics system currently. We do monitor the system, but when go it be to determine what’s going on. KAFKA-2376 is the open JIRA that’s meant to address the underlying connect issue. More databases As we add more databases, we’ll begin to pressure on the two Debezium servers we have. we to the single Debezium that we have into more than one, with some microservices replicating only to one cluster, and the replicating to compatibility checks I in schema section, above, schema registry runs schema compatibility checks box right now. This makes it very easy us to backward and forward incompatible changes from their way into Kafka. We don’t currently have equivalent check at the MySQL layer. This is a problem because it means it’s possible DBA to make incompatible at the MySQL Debezium then when to produce new messages into Kafka. We need to make sure this can’t by adding equivalent checks at the MySQL layer. DBZ-70 discusses Automatic configuration We currently Kafka with topic auto-create with default of 6 partitions, retention. configuration sense Debezium At the very least, should be using log-compaction as retention. We plan to for mis-configured Debezium topics, and updates them to retention settings. Conclusion We’ve been running Debezium production for the past 8 months. Initially, we ran it dark, and enabled it realtime BigQuery pipeline shown in the architecture diagram Recently, we’ve begun consuming messages in microservices and stream processing systems. We look
 is set. This is work we’re planniNg to do shorTly.

MoRe mOniToRing

kafkA connect doesn’t CUrreNtly make it easy to expose MetriCs THrouGH tHe KafKa metriCS fRAMewOrK. AS a rEsULt, There Are Very few Metrics avaIlaBlE from the KafkA CoNnect frameWorK. Debezium doeS exPose meTrIcs VIa JMX (See dBZ-134), buT we areN’T exposIng them to OuR MetricS System cURReNtly. we do monitor the sysTEm, BUt when things go Wrong, iT cAN bE DIffIcuLt to determInE what’S Going on. kaFka-2376 IS thE Open JIRA that’s Meant to addrESs tHe undeRlYinG kafka cOnnecT iSSue.

more databasEs

As We add more MicrosERvice daTAbases, wE’ll begIn tO puT preSSuRe On tHe TWo DEBeZiuM mySqL serverS tHaT we haVe. EvENTUAlly, We pLan tO spliT the single DebEziUm clUSteR that We havE intO mOre thAn one, wIth soMe Microservices rePlicAting only To oNe CluStEr, and THe rest RepLicAting to Others.

UNIfy CoMPATiBility checks

As I menTiONEd In the schEma secTIoN, aBOve, the CoNfLueNt scHEMa regIstrY RuNs schema CompatIBiLiTy checkS oUt of thE bOx rIghT now. THIs maKes it vEry easy fOr us tO Prevent backwarD And forward incOMpATIbLE chaNgeS from making TheiR Way iNto KAFkA. We DOn’t cuRrentLy HAvE An equivalent check at ThE MySQL Layer. this is a probleM because it MEANs it’s posSiblE FoR A DBA to make incoMpatiBle changes AT the MySQl layeR. DebeziuM will then FAIl when trYinG to ProDucE THe New messages inTO kafkA. WE need to MakE sure thIs cAn’t HapPen By Adding equIvalent cHeCkS aT tHe MYSQL lAYer. DBZ-70 diScUssEs ThiS more.

aUtomatIc topIc coNfIgURatIon

We cuRReNTLy ruN KAfKa wiTh tOpIc autO-creATe eNabled wIth a defauLt oF 6 PartItIoNs, and tiMe-based/size-baSeD retention. thIs cOnfiguRATion doesN’t make much sense for DebezIUm topicS. At The veRy leAst, they shOulD be usiNg lOG-compaCtion aS theiR rEteNTIon. We PLAn To wRiTe a script tHAT loOks foR mIs-coNfigureD Debezium topics, and UPdaTes them to apprOprIate RETeNtiON sETtiNgS.

conCLUsion

We’ve been ruNning DebezIuM In Production FOr tHe Past 8 monThs. InitIally, WE ran it dArk, and theN enabled iT fOr thE REalTime BigQueRy pipeliNe shown in THe arcHItEcturE diAgram aBoVe. REcentLy, we’ve BEguN consUming tHe MessagEs in mIcRoservicEs and stream processing syStems. WE look ForWard
 is set. This is work we’r e planning  to d o s hor tl y.
More  monitoring

K a fkaconnect doesn’t curren tly m ak e  ite as y toexposem et r i csth ro ugh  t h eKafka  me trics f ramework.Asaresult, ther e  a re very fe w m etrics avail abl e from  t heK afkacon nectframew o rk. De bezium do es expose metrics v ia  JMX  (see DBZ-134), b u tw e aren’t expos ing th em to o urmet rics syste mcurre n tly. We do m o nit o r the system,  but when t h ing s go w ro ng, it can  be d if f icu lt to deter mine  what’s g oing o n . KAFKA - 2376 is  the o pen  JI RA t h at ’s  me an t  to ad dre s s t he under ly in g Kaf ka c o n n e ct i ssu e.
Moredatabases

As  we  add mor e mic roser vice  d ataba ses, w e’llbe gin to put pres sure  on the t woDe bez iu m MyS Q L serv ers  th at we h ave. Ev e ntu al l y ,  w e plan to split th es i ng le Debez ium cl u st er that weha veinto m ore t hano ne , with s ome mi c ro se rvicesre plicat in g o nly  to o n e cl uster,  and the  rest replicating to others.

Unif y  c o m pa t ibil ity  checks

As  I m e ntio nedi nthe schem a sec ti o n, above, the Confluen tschema  regi stry runs sch ema compat i b i lity che ckso ut of the box rig ht no w. This ma k es it ve ry ea sy for u s to prev e n t backwa rdand  fo rwa r d  i ncompatible c h a nges  f rom mak ing  theirway  in toKaf ka . We don’ t curren tl yha ve  an  equi v alent ch ec k a tthe  MySQ L  layer . Thi s is  a  p r obl em beca u se i t me an sit’s  po ss iblefora  DB A to ma ke incomp ati b le c ha ng es at t he MySQL laye r.  Debeziumwi llthen f a i l when t rying to produce the ne w  messag esintoKafk a. We nee d t o make  su r e this  can’t  happ en  by a dding e qu iva le nt checksa t  th e MyS QL  lay er. DBZ -70 discusses this mor e.

Automatic  to picc o nf igu r at i on
W e  cu r r ently run Kafka  with topi ca ut o-create e n abl ed  with a  defaul t of6  partit ions, and  time-bas ed /siz e - bas ed retenti on. This  configur a tiond oe sn’tmak e much  s ens e for  Debez i umtopic s. Atth e very  leas t,  they sh ould be using log-compa ctionas th eir  retentio n.W e p lan to wr itea script t hat  lo oks f orm is-co nfig u re d D e beziu m to p ics, andu pd ate s th em to appro p r i ate  rete nti o n sett ings .

Conclusion

We ’ ve been runnin g De b e ziu m i n  pro du ction for thepas t8 months.In itially, we  ran itda r k, an d then  enabl ed it f o r  t h e real time  Bi gQuery pi pel in e  shownin  t h e arch itec tu re dia gram a b ove. R ecently, we’ve b egunc o nsumi n g t he me ss ages in micr oservicesand streamproces sing  syst ems. We  l ook fo rwa rd 
 is_set. This_is work we’re planning_to do_shortly.

More_monitoring

Kafka connect_doesn’t_currently make it_easy to expose_metrics through the Kafka_metrics framework. As_a_result, there are very few metrics available from the Kafka connect framework. Debezium does_expose_metrics via_JMX_(see_DBZ-134), but we aren’t exposing_them to our metrics system_currently. We_do monitor the system, but when things go_wrong,_it can be_difficult to determine what’s going on. KAFKA-2376 is the_open JIRA that’s meant to address_the underlying Kafka_connect_issue.

More_databases

As we add more_microservice databases, we’ll begin to put_pressure on the two Debezium MySQL_servers that we have. Eventually, we plan_to split the single Debezium cluster_that we have into more_than one,_with some microservices replicating only_to one cluster,_and the_rest replicating to_others.

Unify compatibility checks

As I mentioned in_the schema section,_above, the Confluent schema registry runs_schema_compatibility checks out_of_the_box right_now. This makes_it_very easy_for_us to prevent backward and forward_incompatible_changes from making their way into Kafka._We don’t currently have_an_equivalent check at the_MySQL layer. This is a_problem because it means it’s possible_for a_DBA to_make incompatible changes at the MySQL layer. Debezium will then fail_when trying to produce the new_messages into Kafka. We_need to_make_sure this can’t_happen_by adding_equivalent checks at the MySQL layer. DBZ-70_discusses this_more.

Automatic topic configuration

We currently run Kafka_with topic auto-create enabled_with_a default of 6 partitions, and_time-based/size-based retention. This configuration doesn’t make_much sense for Debezium topics._At_the_very least, they should be_using log-compaction as their retention. We_plan to write_a script that looks for mis-configured Debezium_topics,_and updates them to appropriate retention_settings.

Conclusion

We’ve_been running Debezium in production for_the_past_8 months. Initially, we ran_it dark, and then enabled it_for the realtime BigQuery pipeline shown in the architecture_diagram above. Recently,_we’ve begun consuming the messages_in_microservices_and stream processing systems. We look forward