anchor.diff:  [[mysql-change-event-values]]
   === Change event values
   
  -
  -
  +The value in a change event is a bit more complicated than the key. Like the key, the value has a `schema` section and a `payload` section. The `schema` section contains the schema that describes the `Envelope` structure of the `payload` section, including its nested fields. Change events for operations that create, update or delete data all have a value payload with an envelope structure. 
  +
  +Consider the same sample table that was used to show an example of a change event key: 
  +
  +[source,sql]
  +----
  +CREATE TABLE customers (
  +  id INTEGER NOT NULL AUTO_INCREMENT PRIMARY KEY,
  +  first_name VARCHAR(255) NOT NULL,
  +  last_name VARCHAR(255) NOT NULL,
  +  email VARCHAR(255) NOT NULL UNIQUE KEY
  +) AUTO_INCREMENT=1001;
  +----
  +
  +ifdef::product[]
  +The value portion of a change event for a change to this table is described for: 
  +
  +* <<mysql-create-events,_create_ events>>
  +* <<mysql-update-events,_update_ events>>
  +* <<mysql-primary-key-updates,Primary key updates>>
  +* <<mysql-delete-events,_delete_ events>>
  +* <<mysql-tombstone-events,Tombstone events>>
  +endif::product[]
  +
  +// Type: continue
  +[id="mysql-create-events"]
  +=== _create_ events
  +
  +The following example shows the value portion of a change event that the connector generates for an operation that creates data in the `customers` table: 
  +
  +[source,json,options="nowrap",subs="+attributes"]
  +----
  +{
  +  "schema": { // <1>
  +    "type": "struct",
  +    "fields": [
  +      {
  +        "type": "struct",
  +        "fields": [
  +          {
  +            "type": "int32",
  +            "optional": false,
  +            "field": "id"
  +          },
  +          {
  +            "type": "string",
  +            "optional": false,
  +            "field": "first_name"
  +          },
  +          {
  +            "type": "string",
  +            "optional": false,
  +            "field": "last_name"
  +          },
  +          {
  +            "type": "string",
  +            "optional": false,
  +            "field": "email"
  +          }
  +        ],
  +        "optional": true,
  +        "name": "mysql-server-1.inventory.customers.Value", // <2>
  +        "field": "before"
  +      },
  +      {
  +        "type": "struct",
  +        "fields": [
  +          {
  +            "type": "int32",
  +            "optional": false,
  +            "field": "id"
  +          },
  +          {
  +            "type": "string",
  +            "optional": false,
  +            "field": "first_name"
  +          },
  +          {
  +            "type": "string",
  +            "optional": false,
  +            "field": "last_name"
  +          },
  +          {
  +            "type": "string",
  +            "optional": false,
  +            "field": "email"
  +          }
  +        ],
  +        "optional": true,
  +        "name": "mysql-server-1.inventory.customers.Value", 
  +        "field": "after"
  +      },
  +      {
  +        "type": "struct",
  +        "fields": [
  +          {
  +            "type": "string",
  +            "optional": false,
  +            "field": "version"
  +          },
  +          {
  +            "type": "string",
  +            "optional": false,
  +            "field": "connector"
  +          },
  +          {
  +            "type": "string",
  +            "optional": false,
  +            "field": "name"
  +          },
  +          {
  +            "type": "int64",
  +            "optional": false,
  +            "field": "ts_sec"
  +          },
  +          {
  +            "type": "boolean",
  +            "optional": true,
  +            "default": false,
  +            "field": "snapshot"
  +          },
  +          {
  +            "type": "string",
  +            "optional": false,
  +            "field": "db"
  +          },
  +          {
  +            "type": "string",
  +            "optional": true,
  +            "field": "table"
  +          },
  +          {
  +            "type": "int64",
  +            "optional": false,
  +            "field": "server_id"
  +          },
  +          {
  +            "type": "string",
  +            "optional": true,
  +            "field": "gtid"
  +          },
  +          {
  +            "type": "string",
  +            "optional": false,
  +            "field": "file"
  +          },
  +          {
  +            "type": "int64",
  +            "optional": false,
  +            "field": "pos"
  +          },
  +          {
  +            "type": "int32",
  +            "optional": false,
  +            "field": "row"
  +          },
  +          {
  +            "type": "int64",
  +            "optional": true,
  +            "field": "thread"
  +          },
  +          {
  +            "type": "string",
  +            "optional": true,
  +            "field": "query"
  +          }
  +        ],
  +        "optional": false,
  +        "name": "io.debezium.connector.mysql.Source", // <3>
  +        "field": "source"
  +      },
  +      {
  +        "type": "string",
  +        "optional": false,
  +        "field": "op"
  +      },
  +      {
  +        "type": "int64",
  +        "optional": true,
  +        "field": "ts_ms"
  +      }
  +    ],
  +    "optional": false,
  +    "name": "mysql-server-1.inventory.customers.Envelope" // <4>
  +  },
  +  "payload": { // <5>
  +    "op": "c", // <6>
  +    "ts_ms": 1465491411815, // <7>
  +    "before": null, // <8>
  +    "after": { // <9>
  +      "id": 1004,
  +      "first_name": "Anne",
  +      "last_name": "Kretchmar",
  +      "email": "annek@noanswer.org"
  +    },
  +    "source": { // <10>
  +      "version": "{debezium-version}",
  +      "connector": "mysql",
  +      "name": "mysql-server-1",
  +      "ts_sec": 0,
  +      "snapshot": false,
  +      "db": "inventory",
  +      "table": "customers",
  +      "server_id": 0,
  +      "gtid": null,
  +      "file": "mysql-bin.000003",
  +      "pos": 154,
  +      "row": 0,
  +      "thread": 7,
  +      "query": "INSERT INTO customers (first_name, last_name, email) VALUES ('Anne', 'Kretchmar', 'annek@noanswer.org')"
  +    }
  +  }
  +}
  +----
  +
  +.Descriptions of _create_ event value fields
  +[cols="1,2,7",options="header"]
  +|===
  +|Item |Field name |Description
  +
  +|1
  +|`schema`
  +|The value's schema, which describes the structure of the value's payload. A change event's value schema is the same in every change event that the connector generates for a particular table. 
  +
  +|2
  +|`name`
  +a|In the `schema` section, each `name` field specifies the schema for a field in the value's payload. +
  + +
  +`mysql-server-1.inventory.customers.Value` is the schema for the payload's `before` and `after` fields. This schema is specific to the `customers` table. +
  + +
  +Names of schemas for `before` and `after` fields are of the form `_logicalName_._tableName_.Value`, which ensures that the schema name is unique in the database. This means that when using the {link-prefix}:{link-avro-serialization}[Avro converter], the resulting Avro schema for each table in each logical source has its own evolution and history.
  +
  +|3
  +|`name`
  +|`io.debezium.connector.mysql.Source` is the schema for the payload's `source` field. This schema is specific to the MySQL connector. The connector uses it for all events that it generates. 
  +
  +|4
  +|`name`
  +|`mysql-server-1.inventory.customers.Envelope` is the schema for the overall structure of the payload, where `mysql-server-1` is the connector name, `inventory` is the database, and `customers` is the table.
  +
  +|5
  +|`payload`
  +|The value's actual data. This is the information that the change event is providing. +
  + +
  +It may appear that the JSON representations of the events are much larger than the rows they describe. This is because the JSON representation must include the schema and the payload portions of the message.
  +However, by using the {link-prefix}:{link-avro-serialization}[Avro converter], you can significantly decrease the size of the messages that the connector streams to Kafka topics.
  +
  +|6
  +|`op`
  +a| Mandatory string that describes the type of operation that caused the connector to generate the event. In this example, `c` indicates that the operation created a row. Valid values are: 
  +
  +* `c` = create
  +* `u` = update
  +* `d` = delete
  +* `r` = read (applies to only snapshots)
  +
  +|7
  +|`ts_ms`
  +a| Optional field that displays the time at which the connector processed the event. The time is based on the system clock in the JVM running the Kafka Connect task.
  +
  +|8
  +|`before`
  +| An optional field that specifies the state of the row before the event occurred. When the `op` field is `c` for create, as it is in this example, the `before` field is `null` since this change event is for new content. 
  +
  +|9
  +|`after`
  +| An optional field that specifies the state of the row after the event occurred. In this example, the `after` field contains the values of the new row's `id`, `first_name`, `last_name`, and `email` columns.
  +
  +|10
  +|`source`
  +a| Mandatory field that describes the source metadata for the event. This field contains information that you can use to compare this event with other events, with regard to the origin of the events, the order in which the events occurred, and whether events were part of the same transaction. The source metadata includes: 
  +
  +* {prodname} version
  +* Connector name
  +* binlog name where the event was recorded
  +* binlog position
  +* Row within the event
  +* If the event was part of a snapshot
  +* Name of the database and table that contain the new row
  +* ID of the MySQL thread that created the event (non-snapshot only)
  +* MySQL server ID (if available)
  +* Timestamp
  +
  +If the {link-prefix}:{link-mysql-connector}#enable-query-log-events[`binlog_rows_query_log_events`] MySQL configuration option is enabled and the connector configuration `include.query` property is enabled, the `source` field also provides the `query` field, which contains the original SQL statement that caused the change event.
  +
  +|===
  +
  +// Type: continue
  +[id="mysql-update-events"]
  +=== _update_ events
  +
  +The value of a change event for an update in the sample `customers` table has the same schema as a _create_ event for that table. Likewise, the event value's payload has the same structure. However, the event value payload contains different values in an _update_ event. Here is an example of a change event value in an event that the connector generates for an update in the `customers` table: 
  +
  +[source,json,options="nowrap",subs="+attributes"]
  +----
  +{
  +  "schema": { ... },
  +  "payload": {
  +    "before": { // <1>
  +      "id": 1004,
  +      "first_name": "Anne",
  +      "last_name": "Kretchmar",
  +      "email": "annek@noanswer.org"
  +    },
  +    "after": { // <2>
  +      "id": 1004,
  +      "first_name": "Anne Marie",
  +      "last_name": "Kretchmar",
  +      "email": "annek@noanswer.org"
  +    },
  +    "source": { // <3>
  +      "version": "{debezium-version}",
  +      "name": "mysql-server-1",
  +      "connector": "mysql",
  +      "name": "mysql-server-1",
  +      "ts_sec": 1465581,
  +      "snapshot": false,
  +      "db": "inventory",
  +      "table": "customers",
  +      "server_id": 223344,
  +      "gtid": null,
  +      "file": "mysql-bin.000003",
  +      "pos": 484,
  +      "row": 0,
  +      "thread": 7,
  +      "query": "UPDATE customers SET first_name='Anne Marie' WHERE id=1004"
  +    },
  +    "op": "u", // <4>
  +    "ts_ms": 1465581029523 
  +  }
  +}
  +----
  +
  +.Descriptions of _update_ event value fields
  +[cols="1,2,7",options="header"]
  +|===
  +|Item |Field name |Description
  +
  +|1
  +|`before`
  +|An optional field that specifies the state of the row before the event occurred. In an _update_ event value, the `before` field contains a field for each table column and the value that was in that column before the database commit. In this example, the `first_name` value is `Anne.`
  +
  +|2
  +|`after`
  +| An optional field that specifies the state of the row after the event occurred. You can compare the `before` and `after` structures to determine what the update to this row was. In the example, the `first_name` value is now `Anne Marie`. 
  +
  +|3
  +|`source`
  +a|Mandatory field that describes the source metadata for the event. The `source` field structure has the same fields as in a _create_ event, but some values are different, for example, the sample _update_ event is from a different position in the binlog. The source metadata includes: 
  +
  +* {prodname} version
  +* Connector name
  +* binlog name where the event was recorded
  +* binlog position
  +* Row within the event
  +* If the event was part of a snapshot
  +* Name of the database and table that contain the updated row
  +* ID of the MySQL thread that created the event (non-snapshot only)
  +* MySQL server ID (if available)
  +* Timestamp
  +
  +If the {link-prefix}:{link-mysql-connector}#enable-query-log-events[`binlog_rows_query_log_events`] MySQL configuration option is enabled and the connector configuration `include.query` property is enabled, the `source` field also provides the `query` field, which contains the original SQL statement that caused the change event.
  +
  +|4
  +|`op`
  +a|Mandatory string that describes the type of operation. In an _update_ event value, the `op` field value is `u`, signifying that this row changed because of an update.
  +
  +|===
  +
  +[NOTE]
  +====
  +Updating the columns for a row's primary/unique key changes the value of the row's key. When a key changes, {prodname} outputs _three_ events: a `DELETE` event and a {link-prefix}:{link-mysql-connector}#mysql-tombstone-events[tombstone event] with the old key for the row, followed by an event with the new key for the row. Details are in the next section. 
  +====
  +
  +// Type: continue
  +[id="mysql-primary-key-updates"]
  +=== Primary key updates
  +
  +An `UPDATE` operation that changes a row's primary key field(s) is known
  +as a primary key change. For a primary key change, in place of an `UPDATE` event record, the connector emits a `DELETE` event record for the old key and a `CREATE` event record for the new (updated) key. These events have the usual structure and content, and in addition, each one has a message header related to the primary key change: 
  +
  +* The `DELETE` event record has `__debezium.newkey` as a message header. The value of this header is the new primary key for the updated row.
  +
  +* The `CREATE` event record has `__debezium.oldkey` as a message header. The value of this header is the previous (old) primary key that the updated row had.
  +
  +// Type: continue
  +[id="mysql-delete-events"]
  +=== _delete_ events
  +
  +The value in a _delete_ change event has the same `schema` portion as _create_ and _update_ events for the same table. The `payload` portion in a _delete_ event for the sample `customers` table looks like this:  
  +
  +[source,json,options="nowrap",subs="+attributes"]
  +----
  +{
  +  "schema": { ... },
  +  "payload": {
  +    "before": { // <1>
  +      "id": 1004,
  +      "first_name": "Anne Marie",
  +      "last_name": "Kretchmar",
  +      "email": "annek@noanswer.org"
  +    },
  +    "after": null, // <2>
  +    "source": { // <3>
  +      "version": "{debezium-version}",
  +      "connector": "mysql",
  +      "name": "mysql-server-1",
  +      "ts_sec": 1465581,
  +      "snapshot": false,
  +      "db": "inventory",
  +      "table": "customers",
  +      "server_id": 223344,
  +      "gtid": null,
  +      "file": "mysql-bin.000003",
  +      "pos": 805,
  +      "row": 0,
  +      "thread": 7,
  +      "query": "DELETE FROM customers WHERE id=1004"
  +    },
  +    "op": "d", // <4>
  +    "ts_ms": 1465581902461 // <5>
  +  }
  +}
  +----
  +
  +.Descriptions of _delete_ event value fields
  +[cols="1,2,7",options="header"]
  +|===
  +|Item |Field name |Description
  +
  +|1
  +|`before`
  +|Optional field that specifies the state of the row before the event occurred. In a _delete_ event value, the `before` field contains the values that were in the row before it was deleted with the database commit.
  +
  +|2
  +|`after`
  +| Optional field that specifies the state of the row after the event occurred. In a _delete_ event value, the `after` field is `null`, signifying that the row no longer exists.
  +
  +|3
  +|`source`
  +a|Mandatory field that describes the source metadata for the event. In a _delete_ event value, the `source` field structure is the same as for _create_ and _update_ events for the same table. Many `source` field values are also the same. In a _delete_ event value, the `ts_ms` and `pos` field values, as well as other values, might have changed. But the `source` field in a _delete_ event value provides the same metadata: 
  +
  +* {prodname} version
  +* Connector name
  +* binlog name where the event was recorded
  +* binlog position
  +* Row within the event
  +* If the event was part of a snapshot
  +* Name of the database and table that contain the updated row
  +* ID of the MySQL thread that created the event (non-snapshot only)
  +* MySQL server ID (if available)
  +* Timestamp
  +
  +If the {link-prefix}:{link-mysql-connector}#enable-query-log-events[`binlog_rows_query_log_events`] MySQL configuration option is enabled and the connector configuration `include.query` property is enabled, the `source` field also provides the `query` field, which contains the original SQL statement that caused the change event.
  +
  +|4
  +|`op`
  +a|Mandatory string that describes the type of operation. The `op` field value is `d`, signifying that this row was deleted.
  +
  +|5
  +|`ts_ms`
  +a|Optional field that displays the time at which the connector processed the event. The time is based on the system clock in the JVM running the Kafka Connect task.
  +
  +|===
  +
  +A _delete_ change event record provides a consumer with the information it needs to process the removal of this row. The old values are included because some consumers might require them in order to properly handle the removal.
  +
  +MySQL connector events are designed to work with link:{link-kafka-docs}/#compaction[Kafka log compaction]. Log compaction enables removal of some older messages as long as at least the most recent message for every key is kept. This lets Kafka reclaim storage space while ensuring that the topic contains a complete data set and can be used for reloading key-based state.
  +
  +// Type: continue
  +[id="mysql-tombstone-events"]
  +.Tombstone events
  +When a row is deleted, the _delete_ event value still works with log compaction, because Kafka can remove all earlier messages that have that same key. However, for Kafka to remove all messages that have that same key, the message value must be `null`. To make this possible, after {prodname}’s MySQL connector emits a _delete_ event, the connector emits a special tombstone event that has the same key but a `null` value.
   
   // Type: reference
   // ModuleID: how-debezium-mysql-connectors-map-data-types
  
positive.code: [[mysql-data-types]]
  == Data type mappings
  
  
  
  ifdef::product[]
  Details are in the following sections:
  
negative.code: [[mysql-change-event-keys]]
  === Change event keys
  
  A change event's key contains the schema for the changed table's key and the changed row's actual key. Both the schema and its corresponding payload contain a field for each column in the changed table's `PRIMARY KEY` (or unique constraint) at the time the connector created the event.
  
  Consider the following `customers` table, which is followed by an example of a change event key for this table. 
  
  .Example table
  [source,sql]
  ----
  CREATE TABLE customers (
    id INTEGER NOT NULL AUTO_INCREMENT PRIMARY KEY,
    first_name VARCHAR(255) NOT NULL,
    last_name VARCHAR(255) NOT NULL,
    email VARCHAR(255) NOT NULL UNIQUE KEY
  ) AUTO_INCREMENT=1001;
  ----
  
  .Example change event key
  Every change event that captures a change to the `customers` table has the same event key schema. For as long as the `customers` table has the previous definition, every change event that captures a change to the `customers` table has the following key structure. In JSON, it looks like this:
  
  [source,json,index=0]
  ----
  {
   "schema": { <1>
      "type": "struct",
      "name": "mysql-server-1.inventory.customers.Key", <2>
      "optional": false, <3>
      "fields": [ <4>
        {
          "field": "id",
          "type": "int32",
          "optional": false
        }
      ]
    },
   "payload": { <5>
      "id": 1001
    }
  }
  ----
  
  .Description of change event key
  [cols="1,2,7",options="header"]
  |===
  |Item |Field name |Description
  
  |1
  |`schema`
  |The schema portion of the key specifies a Kafka Connect schema that describes what is in the key's `payload` portion. 
  
  |2
  |`mysql-server-1.inventory.customers.Key`
  a|Name of the schema that defines the structure of the key's payload. This schema describes the structure of the primary key for the table that was changed. Key schema names have the format _connector-name_._database-name_._table-name_.`Key`. In this example: + 
  
  * `mysql-server-1` is the name of the connector that generated this event. + 
  * `inventory` is the database that contains the table that was changed. +
  * `customers` is the table that was updated.
  
  |3
  |`optional`
  |Indicates whether the event key must contain a value in its `payload` field. In this example, a value in the key's payload is required. A value in the key's payload field is optional when a table does not have a primary key.
  
  |4
  |`fields` 
  |Specifies each field that is expected in the `payload`, including each field's name, type, and whether it is required.
  
  |5
  |`payload`
  |Contains the key for the row for which this change event was generated. In this example, the key, contains a single `id` field whose value is `1001`.
  
  |===
  
  // Type: concept
  // ModuleID: about-values-in-debezium-mysql-change-events
  