anchor.diff:  = Distributed Tracing with Debezium
   
   The current pattern in application development gravitates toward microservices and microservices architecture.
  -While this approach gives the developer teams great flexibility in independent deployment and development velocity the drawback is at hand when you try to track a bug in production.
  +While this approach gives the developer teams great flexibility in terms of independent deployments and development velocity, the drawback is at hand when you try to track a bug in production.
   
  -Monolithic applications sit nicely at a single place so you can introspect the code-flows and their runtime state.
  -This is a problem with microservice architecture as a single business transaction can span across tens of services deployed in separate processes and nodes.
  +Monolithic applications sit nicely at a single place so you can introspect the code flows and the application's runtime state.
  +This is more challenging with microservice architectures, as a single business transaction can span across tens of services deployed in separate processes and compute nodes.
   
  -You can rely on traditional methods like logging where you need to collect and correlate logs at a single place so you can try to reconstruct the business transaction path.
  -This is one of the tool in the toolbox we need to use but it still can be crude and it will not provide the necessary context.
  +You can rely on traditional methods like logging where you need to collect and correlate logs at a single place, so you can try to reconstruct the business transaction path.
  +This is one of the tools in the box we can use, but it still can be crude and it will not provide all the necessary context.
   https://microservices.io/patterns/observability/distributed-tracing.html[Distributed Tracing] comes here to the rescue.
   
   == Distributed Tracing
   
  -Distributed tracing allows services to leave breadcrumbs during the execution with enough information to create an execution path of the business transaction enriched with contextual data like who, what,  and where.
  -A developer can then use it to browse through the recorded executions and check for errors or anomalies in execution that can signify either problem in deployment (services unavailability) or even a bug.
  +Distributed tracing allows services to leave breadcrumbs during the execution with enough information to create an execution path of the business transaction enriched with contextual data like "who", "what", and "where".
  +SRE teams and developers can then use it to browse through the recorded executions and check for errors or anomalies in execution that can signify either problems with deployments (services unavailabe) or even bugs.
  +
  +And this is where Debezium becomes part of the picture.
  +Data change events, as captured by Debezium from a database, and propagated via Kafka Connect and Apache Kafka to one more more downstream consumers are part of a data flow which is very valuable to have insight into.
  +How long does it take for change events to flow from source database to sink systems?
  +Where is the most time spent in the pipeline?
  +Are there any anomalies like spikes in end-to-end lags?
  +The integration of distributed tracing with Debezium can help to answer these questions.
   
   == OpenTracing
   
  -There are multiple solutions for distributed tracing but we have decided to follow and use open standard specification https://opentracing.io/[OpenTracing].
  +There are multiple solutions for distributed tracing, but as a starting poin we have decided to follow and use the https://opentracing.io/[OpenTracing] specification.
   OpenTracing is an incubating project of https://www.cncf.io/[Cloud Native Computing Foundation] which guarantees that the user will be free of any vendor lock-in by adhering to an open standard.
   
   [NOTE]
   ====
  -OpenTracing project will in future be replaced with an improved https://opentelemetry.io/[OpenTelemtry] standard.
  -Debezium will use it in the future too.
  +The OpenTracing project is in the process of being merged with OpenCensus to the improved https://opentelemetry.io/[OpenTelemetry] standard.
  +Debezium uses OpenTracing at this point for alignment reasons with other projects (e.g. Quarkus),
  +but it will use and support OpenTelemetry in the future, too.
   ====
   
   A distributed trace in OpenTracing consists of a set of spans.
  
positive.code: The spans can form a tree when a larger part of the business transaction represented by one span can be compounded of multiple tasks represented by additional spans that have a parent-child relationship to the main span.
  
  OpenTracing is only the specification.
  To use it you need to have an implementation too.
  Debezium could use any OpenTracing client implementation but examples and documentation is based on https://www.jaegertracing.io/[Jaeger] distributed tracing.
  
  Jaeger consists of multiple components responsible for data collection and storage and also a graphical user interface available as a web application.
  https://www.jaegertracing.io/docs/1.21/getting-started/#all-in-one[All In One] container image will be used to simplify the deployment.
  
  == Debezium and OpenTracing
  
  Debezium integration with OpenTracing consists of three distinct components
  
  * `EventDispatcher` in Debezium outbox extension
  * `EventRouter` SMT
  * `ActivateTracingSpan` SMT
  
  The first two must be used hand-in-hand when a service using the outbox pattern should be traced.
  The third one is intended for general use.
  
  === Outbox Distributed Tracing
  
  The biggest problem with tracing integration is keeping the trace across process boundaries so the additional spans are recorded in the same trace to enable end-to-end tracing.
  Opentracing specification provides a way how to export and import trace related metadata so the trace could be passed among different processes.
  In the outbox extension we use this approach to export the metadata into a new database column in the outbox table and then event router SMT imports them and resumes the trace.
  
  In each of the steps executed one or more spans are created
  
  * When an event arrives at `EventDispatcher` a new span `outbox-write` is created.
  It is created as a child of a current active span or as a root span if no parent span is available.
  * The span metadata is exported into a distinct field of the outbox event.
  * Outbox event is written to and deleted from the outbox table.
  * Event Router SMT receives the event and imports the span metadata from the field
  * Two new spans are created
  ** `db-log-write` with start timestamp to database write timestamp.
  Fields from the `source` block are added to the span as *tags*.
  ** `debezium-read` with start time set to the processing timestamp.
  Fields from the envelope are added to the span as *tags*.
  * Optionally if OpenTracing integration is enabled at the Kafka producer level a new span is created by the Kafka producer representing the write of the message to a Kafka topic with relevant metadata.
  
  
  ==== Demo
  
  The https://github.com/debezium/debezium-examples/tree/master/outbox[outbox example] was extended with distributed tracing support to demonstrate the functionality.
  
  [NOTE]
  ====
  This demo uses https://strimzi.io/[Strimzi] container image for Kafka Connect as it already contains baked-in integration of OpenTracing at Kafka producer level.
  ====
  
  To try it yourself you need to:
  
  * checkout the repository and switch to the `outbox` directory
  * build the services
  ----
  $ mvn clean install
  ----
  * deploy application
  ----
  export DEBEZIUM_VERSION=1.4
  docker-compose up --build
  ----
  * register Debezium connector to listen on an outbox table
  ----
  $ http PUT http://localhost:8083/connectors/outbox-connector/config < register-postgres.json
  HTTP/1.1 201 Created
  
negative.code: </div>
  ++++
  
  Click on the `addOrder` service.
  A tree will open that displays how the initial request incoming via REST API was
  
  * written to the database by the outbox extension
  * read by Debezium and processed by outbox SMT
  * written to a Kafka topic
  * read from a Kafka topic by `shipment-service`
  * processed in the different `shipment-service` business methods
  ++++
  <div class="imageblock centered-image">
      <img src="/assets/images/tracing-tutorial/tracing-trace.png" class="responsive-image" alt="Service traces">
  </div>
  ++++
  
  Click on the `db-log-write` and `debezium-read` spans.
  The *tags* of each of them contain extracted Debezium-related metadata like `operation` or `source` fields:
  ++++
  <div class="imageblock centered-image">
      <img src="/assets/images/tracing-tutorial/tracing-debezium-details.png" class="responsive-image" alt="Service traces">
  