How to synchronize Elasticsearch with MySQL
Problem Statement

A common scenario for tech companies is to start building their core business functionalities around one or more databases, and then start connecting services to those databases to perform searches on text data, such as searching for a street name in the user addresses column of a users table, or searching for book titles and author names in the catalog of a library.

So far everything is fine. The company keeps growing and acquires more and more customers. But then searches start to become slower and slower, which tends to irritate their customers and makes it hard to convince new prospects to buy in.

Looking under the hood, the engineers realize that MySQL’s FULLTEXT indexes are not ideal for text look-ups on large datasets, and in order to scale up their operations, the engineers decide to move on to a more dedicated and battle-tested text search engine.

Enters Elasticsearch and its underlying Lucene search engine. Elasticsearch indexes data using an inverted document index, and this results in a blazing-fast full-text search.

A new challenge then comes in: How to get the data that is in a MySQL database into an Elasticsearch index, and how to keep the latter synchronized with the former?

Solution

The modern data plumber’s toolkit contains a plethora of software for any data manipulation task. In this article, we’ll focus on Logstash from the ELK stack in order to periodically fetch data from MySQL, and mirror it on Elasticsearch. This will take into consideration any changes on the MySQL data records, such as create , update , and delete , and have the same replicated on Elasticsearch documents.

Benefits from using Logstash

The problem we are trying to solve here — sending data periodically from MySQL to Elasticsearch for syncing the two — can be solved with a Shell or Python script ran by a cron job or any job scheduler, BUT this would deprive us from the benefits otherwise acquired by configuring Logstash and its plugin-based setup:

Although we focus on MySQL here, Logstash can ingest data from many sources, making it a centralized data input node.

Logstash offers the possibility to parse, transform, and filter data on the fly, as it passes from source to destination.

As for the input sources, there are many output destinations available — Elasticsearch being the go-to output.

As part of the ELK Stack, everything will come together nice and smoothly later with Elasticsearch and Kibana for metrics and visualization.

Use Cases

We will cover two scenarios in the following steps:

Creating an Elasticsearch index and indexing database records from scratch. Incremental update of the Elasticsearch index based on changes occurring on the database records (creation, update, deletion).

Implementing a proof of concept

Imagine you are opening an online library where avid readers can search your catalog of books to find their next reading. Your catalog contains millions of titles, ranging from scientific literature volumes to pamphlets about exotic adventures.

We will create an initial database table books with a few thousands of records with book titles, authors, ISBN and publication date. We’ll use the Goodreads book catalog that can be found on Kaggle. This initial table will serve prototyping the use case (1) Building an index from scratch.

We will create triggers on the table books that will populate a journal table books_journal with all changes on the books table (e.g. create , update , delete ). This means that whenever a record on the books table is created, updated, or deleted, this action will be recorded on the books_journal table, and the same action will be done on the corresponding Elasticsearch document. This will serve prototyping the use case (2) Incremental update of the Elasticsearch index.

We’ll prototype and test the above concepts by defining a micro-services architecture using docker-compose.

Architecture of this project — Image by Author

Prerequisites

Steps

Full source code can be found on GitHub at sync-elasticsearch-mysql.

Start by creating a directory to host this project (named e.g. sync-elasticsearch-mysql ) and create a docker-compose.yaml file inside that directory with the following initial setup:

version: "3"

services:

...

# Here will come the services definition

...

2. Setup a MySQL database: Create a directory data/ where we’ll store MySQL dump files with the pre-cleaned books data for the books table, as well as the triggers for the books table (on create, update, and delete), and a new_arrivals table with books we will add to our catalog to simulate new records, and the table books_journal .

You can find these dump files in the project repository. Please copy them to the data/ directory so that the MySQL container will add them to the books database during the startup process.

version: "3"

services:

# add this:

mysql:

image: mysql:8

container_name: sem_mysql

ports:

- 3306:3306

environment:

MYSQL_RANDOM_ROOT_PASSWORD: "yes"

MYSQL_DATABASE: books

MYSQL_USER: avid_reader

MYSQL_PASSWORD: i_love_books

volumes:

# Dump files for initiating tables

- ./data/:/docker-entrypoint-initdb.d/

Note that storing usernames and passwords in plain text in your source code is not advised. The above is done for the sake of prototyping this project only and is not in any case a recommended practice.

To start MySQL and check that data has been added successfully, from you project’s directory run on a terminal:

docker-compose up -d mysql # -d is for detached mode # Once container started, log into the MySQL container

docker exec -it sem_mysql bash # Start a MySQL prompt

mysql -uavid_reader -pi_love_books # Check that tables have been loaded from dump files

use books;

show tables;

Listing of the available tables and a sample of the data — Image by Author

To show the triggers with a JSON-like formatting, use command:

SHOW TRIGGERS \G; # To exit the MySQL prompt press CTRL+D

# To logout from the MySQL container, press CTRL+D

Now that we have our database properly set up, we can move on to the meat and potatoes of this project.

3. Setup Elasticsearch and Kibana: To setup Elasticsearch (without indexing any document yet), add this to your docker-compose.yaml file:

Note that the volumes definition is recommended in order to mount the Elasticsearch index data from the docker volume to your directory file system.

To get Elasticsearch and Kibana started, run the following from your project directory:

docker-compose up -d elasticsearch kibana # -d is for detached mode # To check if everything so far is running as it should, run:

docker-compose ps

After executing the above commands, 3 containers should be up and running — Image by Author

Let’s check if there is any index in our Elasticsearch node so far:

# On your terminal, log into the Elasticsearch container

docker exec -it sem_elasticsearch bash # Once logged in, use curl to request a listing of the indexes

curl localhost:9200/_cat/indices

If Kibana is up and running, you’ll see a list of indices used for metrics and visualization, but nothing related to our books yet — and if Kibana hasn’t been started, the list of indices will be empty.

List of indexes created by Kibana — Image by Author

4. Setup Logstash to pipe data from MySQL to Elasticsearch:

To connect Logstash to MySQL, we will use the official JDBC driver available at this address.

Let’s create a Dockerfile (named Dockerfile-logstash in the same directory) to pull a Logstash image, download the JDBC connector, and start a Logstash container. Add these lines to your Dockerfile:

FROM docker.elastic.co/logstash/logstash:7.9.3

RUN curl -L --output "mysql-connector-java-8.0.22.tar.gz" "

&& tar -xf "mysql-connector-java-8.0.22.tar.gz" "mysql-connector-java-8.0.22/mysql-connector-java-8.0.22.jar" \

&& mv "mysql-connector-java-8.0.22/mysql-connector-java-8.0.22.jar" "mysql-connector-java-8.0.22.jar" \

&& rm -r "mysql-connector-java-8.0.22" "mysql-connector-java-8.0.22.tar.gz" # Download JDBC connector for LogstashRUN curl -L --output "mysql-connector-java-8.0.22.tar.gz" " https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-8.0.22.tar.gz " \&& tar -xf "mysql-connector-java-8.0.22.tar.gz" "mysql-connector-java-8.0.22/mysql-connector-java-8.0.22.jar" \&& mv "mysql-connector-java-8.0.22/mysql-connector-java-8.0.22.jar" "mysql-connector-java-8.0.22.jar" \&& rm -r "mysql-connector-java-8.0.22" "mysql-connector-java-8.0.22.tar.gz" ENTRYPOINT ["/usr/local/bin/docker-entrypoint"]

Then add the following snippet to your docker-compose.yaml file:

version: "3"

services:

...

logstash:

build:

context: .

dockerfile: Dockerfile-logstash

container_name: sem_logstash

depends_on:

- mysql

- elasticsearch

volumes:

# We will explain why and how to add volumes below

Logstash uses defined pipelines to know where to get data from, how to filter it, and where should it go. We will define two pipelines: one for creating an Elasticsearch index from scratch (first scenario), and one for incremental updates on changes to the database records (second scenario).

Please check documentation for an explanation of each of the fields used in the pipeline definition:

4.a. First Scenario — Creating an Elasticsearch index from scratch:

In your project directory, create a volumes folder (if not already created) then create a directory to host our Logstash configuration:

mkdir -p volumes/logstash/config

Then in this config directory, create a file pipelines.yml containing:

- pipeline.id: from-scratch-pipeline

path.config: "/usr/share/logstash/pipeline/from-scratch.conf"

We then create a folder pipeline to host our pipelines definitions:

mkdir volumes/logstash/pipeline

and there create a file from-scratch.conf with the following content:

input {

jdbc {

jdbc_driver_library => "/usr/share/logstash/mysql-connector-java-8.0.22.jar"

jdbc_driver_class => "com.mysql.jdbc.Driver"

jdbc_connection_string => "jdbc:mysql://mysql:3306"

jdbc_user => "avid_reader"

jdbc_password => "i_love_books"

clean_run => true

record_last_run => false

statement_filepath => "/usr/share/logstash/config/queries/from-scratch.sql"

}

}

mutate {

remove_field => ["

}

} filter {mutate {remove_field => [" @version ", " @timestamp "]

# stdout { codec => rubydebug { metadata => true } }

elasticsearch {

hosts => ["

index => "books"

action => "index"

document_id => "%{isbn}"

}

} output {# stdout { codec => rubydebug { metadata => true } }elasticsearch {hosts => [" http://elasticsearch:9200 "]index => "books"action => "index"document_id => "%{isbn}"

We define where to find the JDBC connector in jdbc_driver_library , setup where to find MySQL in jdbc_connection_string , instruct the plugin to run from scratch in clean_run , and we define where to find the SQL statement to fetch and format the data records in statement_filepath .

The filter basically removes extra fields added by the plugin.

In the output, we define where to find the Elasticsearch host, set the name of the index to books (can be a new or an existing index), define which action to perform (can be index , create , update , delete — see docs), and setup which field will serve as a unique ID in the books index — ISBN is an internationally unique ID for books.

To add the SQL query referenced by statement_filepath , let’s create a folder queries :

mkdir volumes/logstash/config/queries/

Then add file from-scratch.sql with as little as:

SELECT * FROM books.books

to get our index built with all the records available so far in our books table. Note that the query should NOT end with a semi-colon (;).

Now to mount the configuration files and definitions created above, add the following to your docker-compose.yaml file:

version: "3"

services:

...

logstash:

...

volumes:

- ./volumes/logstash/pipeline/:/usr/share/logstash/pipeline/

- ./volumes/logstash/config/pipelines.yml:/usr/share/logstash/config/pipelines.yml

- ./volumes/logstash/config/queries/:/usr/share/logstash/config/queries/

Note that volumes directives are a way to tell Docker to mount a directory or file inside the container (right hand side of the :) onto a directory or file on your machine or server (left hand side of the :). Checkout the official Docker Compose documentation on volumes .

Testing

In order to test the implementation for this first scenario, run on your terminal:

docker-compose up logstash

If the run is successful, you will see a message similar to Logstash shut down and the container will exit with error code 0.
['Redouane Achouri']
['Logstash', 'MySQL', 'Elasticsearch', 'Data Science', 'Data Engineering']