Chapter 2. Getting started
Chapter 2. Getting started 2.1. AMQ Streams distribution AMQ Streams is distributed as single ZIP file. This ZIP file contains AMQ Streams components: Apache ZooKeeper Apache Kafka Apache Kafka Connect Apache Kafka MirrorMaker Kafka Bridge Kafka Exporter 2.2. Downloading an AMQ Streams Archive An archived distribution of AMQ Streams is available for download from the Red Hat website. You can download a copy of the distribution by following the steps below. Procedure Download the latest version of the Red Hat AMQ Streams archive from the AMQ Streams software downloads page . 2.3. Installing AMQ Streams Follow this procedure to install the latest version of AMQ Streams on Red Hat Enterprise Linux. For instructions on upgrading an existing cluster to AMQ Streams 2.1, see AMQ Streams and Kafka upgrades . Prerequisites Download the installation archive . Review the Supported configurations . Procedure Add new kafka user and group. sudo groupadd kafka sudo useradd -g kafka kafka sudo passwd kafka Create directory /opt/kafka . sudo mkdir /opt/kafka Create a temporary directory and extract the contents of the AMQ Streams ZIP file. mkdir /tmp/kafka unzip amq-streams_y.y-x.x.x.zip -d /tmp/kafka Move the extracted contents into /opt/kafka directory and delete the temporary directory. sudo mv /tmp/kafka/ kafka_y.y-x.x.x /* /opt/kafka/ rm -r /tmp/kafka Change the ownership of the /opt/kafka directory to the kafka user. sudo chown -R kafka:kafka /opt/kafka Create directory /var/lib/zookeeper for storing ZooKeeper data and set its ownership to the kafka user. sudo mkdir /var/lib/zookeeper sudo chown -R kafka:kafka /var/lib/zookeeper Create directory /var/lib/kafka for storing Kafka data and set its ownership to the kafka user. sudo mkdir /var/lib/kafka sudo chown -R kafka:kafka /var/lib/kafka 2.4. Data storage considerations An efficient data storage infrastructure is essential to the optimal performance of AMQ Streams. AMQ Streams requires block storage and works well with cloud-based block storage solutions, such as Amazon Elastic Block Store (EBS). The use of file storage is not recommended. Choose local storage when possible. If local storage is not available, you can use a Storage Area Network (SAN) accessed by a protocol such as Fibre Channel or iSCSI. 2.4.1. Apache Kafka and ZooKeeper storage support Use separate disks for Apache Kafka and ZooKeeper. Kafka supports JBOD (Just a Bunch of Disks) storage, a data storage configuration of multiple disks or volumes. JBOD provides increased data storage for Kafka brokers. It can also improve performance. Solid-state drives (SSDs), though not essential, can improve the performance of Kafka in large clusters where data is sent to and received from multiple topics asynchronously. SSDs are particularly effective with ZooKeeper, which requires fast, low latency data access. Note You do not need to provision replicated storage because Kafka and ZooKeeper both have built-in data replication. 2.4.2. File systems It is recommended that you configure your storage system to use the XFS file system. AMQ Streams is also compatible with the ext4 file system, but this might require additional configuration for best results. Additional resources For more information about XFS, see The XFS File System . 2.5. Running a single node AMQ Streams cluster This procedure shows how to run a basic AMQ Streams cluster consisting of a single Apache ZooKeeper node and a single Apache Kafka node, both running on the same host. The default configuration files are used for ZooKeeper and Kafka. Warning A single node AMQ Streams cluster does not provide reliability and high availability and is suitable only for development purposes. Prerequisites AMQ Streams is installed on the host Running the cluster Edit the ZooKeeper configuration file /opt/kafka/config/zookeeper.properties . Set the dataDir option to /var/lib/zookeeper/ : dataDir=/var/lib/zookeeper/ Edit the Kafka configuration file /opt/kafka/config/server.properties . Set the log.dirs option to /var/lib/kafka/ : log.dirs=/var/lib/kafka/ Switch to the kafka user: su - kafka Start ZooKeeper: /opt/kafka/bin/zookeeper-server-start.sh -daemon /opt/kafka/config/zookeeper.properties Check that ZooKeeper is running: jcmd | grep zookeeper Returns: number org.apache.zookeeper.server.quorum.QuorumPeerMain /opt/kafka/config/zookeeper.properties Start Kafka: /opt/kafka/bin/kafka-server-start.sh -daemon /opt/kafka/config/server.properties Check that Kafka is running: jcmd | grep kafka Returns: number kafka.Kafka /opt/kafka/config/server.properties Additional resources For more information about installing AMQ Streams, see Section 2.3, "Installing AMQ Streams" . For more information about configuring AMQ Streams, see Section 2.8, "Configuring AMQ Streams" . 2.6. Using the cluster This procedure describes how to start the Kafka console producer and consumer clients and use them to send and receive several messages. A new topic is automatically created in step one. Topic auto-creation is controlled using the auto.create.topics.enable configuration property (set to true by default). Alternatively, you can configure and create topics before using the cluster. For more information, see Topics . Prerequisites AMQ Streams is installed on the host ZooKeeper and Kafka are running Procedure Start the Kafka console producer and configure it to send messages to a new topic: /opt/kafka/bin/kafka-console-producer.sh --broker-list <bootstrap-address> --topic <topic-name> For example: /opt/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-topic Enter several messages into the console. Press Enter to send each individual message to your new topic: >message 1 >message 2 >message 3 >message 4 When Kafka creates a new topic automatically, you might receive a warning that the topic does not exist: WARN Error while fetching metadata with correlation id 39 : {4-3-16-topic1=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) The warning should not reappear after you send further messages. In a new terminal window, start the Kafka console consumer and configure it to read messages from the beginning of your new topic. /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server <bootstrap-address> --topic <topic-name> --from-beginning For example: /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-topic --from-beginning The incoming messages display in the consumer console. Switch to the producer console and send additional messages. Check that they display in the consumer console. Stop the Kafka console producer and then the consumer by pressing Ctrl+C . 2.7. Stopping the AMQ Streams services You can stop the Kafka and ZooKeeper services by running a script. All connections to the Kafka and ZooKeeper services will be terminated. Prerequisites AMQ Streams is installed on the host ZooKeeper and Kafka are up and running Procedure Stop the Kafka broker. su - kafka /opt/kafka/bin/kafka-server-stop.sh Confirm that the Kafka broker is stopped. jcmd | grep kafka Stop ZooKeeper. su - kafka /opt/kafka/bin/zookeeper-server-stop.sh 2.8. Configuring AMQ Streams Prerequisites AMQ Streams is downloaded and installed on the host Procedure Open ZooKeeper and Kafka broker configuration files in a text editor. The configuration files are located at : ZooKeeper /opt/kafka/config/zookeeper.properties Kafka /opt/kafka/config/server.properties Edit the configuration options. The configuration files are in the Java properties format. Every configuration option should be on separate line in the following format: Lines starting with # or ! will be treated as comments and will be ignored by AMQ Streams components. Values can be split into multiple lines by using \ directly before the newline / carriage return. Save the changes Restart the ZooKeeper or Kafka broker Repeat this procedure on all the nodes of the cluster. 2.9. Loading configuration values from environment variables Use the Environment Variables Configuration Provider plugin to load configuration data from environment variables. You can use the Environment Variables Configuration Provider, for example, to load certificates or JAAS configuration from environment variables. You can use the provider to load configuration data for all Kafka components, including producers and consumers. Use the provider, for example, to provide the credentials for Kafka Connect connector configuration. Prerequisites AMQ Streams is downloaded and installed on the host The Environment Variables Configuration Provider JAR file from the AMQ Streams archive Procedure Add the Environment Variables Configuration Provider JAR file to the Kafka libs directory. Initialize the Environment Variables Configuration Provider in the configuration properties file of the Kafka component. For example, to initialize the provider for Kafka, add the configuration to the server.properties file. Configuration to enable the Environment Variables Configuration Provider config.providers=env config.providers.env.class=io.strimzi.kafka.EnvVarConfigProvider Add configuration to the properties file to load data from environment variables. Configuration to load data from an environment variable option=USD{env: MY_ENV_VAR_NAME } Use capitalized or upper-case environment variable naming conventions, such as MY_ENV_VAR_NAME . Save the changes. Restart the Kafka component.
sudo groupadd kafka sudo useradd -g kafka kafka sudo passwd kafka sudo mkdir /opt/kafka mkdir /tmp/kafka unzip amq-streams_y.y-x.x.x.zip -d /tmp/kafka sudo mv /tmp/kafka/ kafka_y.y-x.x.x /* /opt/kafka/ rm -r /tmp/kafka sudo chown -R kafka:kafka /opt/kafka sudo mkdir /var/lib/zookeeper sudo chown -R kafka:kafka /var/lib/zookeeper sudo mkdir /var/lib/kafka sudo chown -R kafka:kafka /var/lib/kafka dataDir=/var/lib/zookeeper/ log.dirs=/var/lib/kafka/ su - kafka /opt/kafka/bin/zookeeper-server-start.sh -daemon /opt/kafka/config/zookeeper.properties jcmd | grep zookeeper number org.apache.zookeeper.server.quorum.QuorumPeerMain /opt/kafka/config/zookeeper.properties /opt/kafka/bin/kafka-server-start.sh -daemon /opt/kafka/config/server.properties jcmd | grep kafka number kafka.Kafka /opt/kafka/config/server.properties /opt/kafka/bin/kafka-console-producer.sh --broker-list <bootstrap-address> --topic <topic-name> /opt/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-topic >message 1 >message 2 >message 3 >message 4 WARN Error while fetching metadata with correlation id 39 : {4-3-16-topic1=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server <bootstrap-address> --topic <topic-name> --from-beginning /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-topic --from-beginning su - kafka /opt/kafka/bin/kafka-server-stop.sh jcmd | grep kafka su - kafka /opt/kafka/bin/zookeeper-server-stop.sh <option> = <value> This is a comment sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="bob" password="bobs-password"; config.providers=env config.providers.env.class=io.strimzi.kafka.EnvVarConfigProvider option=USD{env: MY_ENV_VAR_NAME }