Chapter 2. Evaluate AMQ Streams
Chapter 2. Evaluate AMQ Streams The procedures in this chapter provide a quick way to evaluate the functionality of AMQ Streams. Follow the steps in the order provided to install AMQ Streams, and start sending and receiving messages from a topic: Ensure you have the required prerequisites Install AMQ Streams Create a Kafka cluster Enable authentication for secure access to the Kafka cluster Access the Kafka cluster to send and receive messages Ensure you have the prerequisites and then follow the tasks in the order provided in this chapter. 2.1. Prerequisites An OpenShift Container Platform cluster (4.6 and 4.8) running on which to deploy AMQ Streams must be running. You need to be able to access the AMQ Streams download site . 2.2. Downloading AMQ Streams A ZIP file contains the resources required for installation of AMQ Streams, along with examples for configuration. Procedure Ensure your subscription has been activated and your system is registered. For more information about using the Customer Portal to activate your Red Hat subscription and register your system for packages, see Appendix A, Using your subscription . Download the amq-streams-x.y.z-ocp-install-examples.zip file from the AMQ Streams download site . Unzip the file to any destination. Windows or Mac: Extract the contents of the ZIP archive by double clicking on the ZIP file. Red Hat Enterprise Linux: Open a terminal window in the target machine and navigate to where the ZIP file was downloaded. Extract the ZIP file with this command: unzip amq-streams-x.y.z-ocp-install-examples.zip 2.3. Installing AMQ Streams You install AMQ Streams with the Custom Resource Definitions (CRDs) required for deployment. In this task you create namespaces in the cluster for your deployment. It is good practice to use namespaces to separate functions. Prerequisites Installation requires a user with cluster-admin role, such as system:admin . Procedure Login in to the OpenShift cluster using an account that has cluster admin privileges. For example: oc login -u system:admin Create a new kafka (project) namespace for the AMQ Streams Kafka Cluster Operator. oc new-project kafka Modify the installation files to reference the new kafka namespace where you will install the AMQ Streams Kafka Cluster Operator. Note By default, the files work in the myproject namespace. On Linux, use: sed -i 's/namespace: .*/namespace: kafka/' install/cluster-operator/*RoleBinding*.yaml On Mac, use: sed -i '' 's/namespace: .*/namespace: kafka/' install/cluster-operator/*RoleBinding*.yaml Deploy the CRDs and role-based access control (RBAC) resources to manage the CRDs. oc project kafka oc apply -f install/cluster-operator/ Create a new my-kafka-project namespace where you will deploy your Kafka cluster. oc new-project my-kafka-project Give access to my-kafka-project to a non-admin user developer . For example: oc adm policy add-role-to-user admin developer -n my-kafka-project Set the value of the STRIMZI_NAMESPACE environment variable to give permission to the Cluster Operator to watch the my-kafka-project namespace. oc set env deploy/strimzi-cluster-operator STRIMZI_NAMESPACE=kafka,my-kafka-project -n kafka oc apply -f install/cluster-operator/020-RoleBinding-strimzi-cluster-operator.yaml -n my-kafka-project oc apply -f install/cluster-operator/032-RoleBinding-strimzi-cluster-operator-topic-operator-delegation.yaml -n my-kafka-project oc apply -f install/cluster-operator/031-RoleBinding-strimzi-cluster-operator-entity-operator-delegation.yaml -n my-kafka-project The commands create role bindings that grant permission for the Cluster Operator to access the Kafka cluster. Create a new cluster role strimzi-admin . oc apply -f install/strimzi-admin Add the role to the non-admin user developer . oc adm policy add-cluster-role-to-user strimzi-admin developer 2.4. Creating a cluster With AMQ Streams installed, you create a Kafka cluster, then a topic within the cluster. When you create a cluster, the Cluster Operator you deployed when installing AMQ Streams watches for new Kafka resources. Prerequisites For the Kafka cluster, ensure a Cluster Operator is deployed. For the topic, you must have a running Kafka cluster. Procedure Log in to the my-kafka-project namespace as user developer . For example: After new users log in to OpenShift Container Platform, an account is created for that user. Create a new my-cluster Kafka cluster with 3 Zookeeper and 3 broker nodes. Use ephemeral storage Expose the Kafka cluster outside of the OpenShift cluster using an external listener configured to use route . cat << EOF | oc create -f - apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: name: my-cluster spec: kafka: replicas: 3 listeners: - name: plain port: 9092 type: internal tls: false - name: tls port: 9093 type: internal tls: true - name: external port: 9094 type: route 1 tls: true storage: type: ephemeral zookeeper: replicas: 3 storage: type: ephemeral entityOperator: topicOperator: {} EOF Wait for the cluster to be deployed: oc wait my-kafka-project/my-cluster --for=condition=Ready --timeout=300s -n kafka When your cluster is ready, create a topic to publish and subscribe from your external client. Create the following my-topic custom resource definition with 3 replicas and 3 partitions in the my-cluster Kafka cluster: cat << EOF | oc create -f - apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaTopic metadata: name: my-topic labels: strimzi.io/cluster: "my-cluster" spec: partitions: 3 replicas: 3 EOF 2.5. Accessing the cluster As route is used for external access to the cluster, a cluster CA certificate is required to enable TLS (Transport Layer Security) encryption between the broker and the client. Prerequisites You need a Kafka cluster running within the OpenShift cluster. The Cluster Operator must also be running. Procedure Find the address of the bootstrap route : oc get routes my-cluster-kafka-bootstrap -o=jsonpath='{.status.ingress[0].host}{"\n"}' Use the address together with port 443 in your Kafka client as the bootstrap address. Extract the public certificate of the broker certification authority: oc extract secret/my-cluster-cluster-ca-cert --keys=ca.crt --to=- > ca.crt Import the trusted certificate to a truststore: keytool -keystore client.truststore.jks -alias CARoot -import -file ca.crt You are now ready to start sending and receiving messages. 2.6. Sending and receiving messages from a topic You can test your AMQ Streams installation by sending and receiving messages outside the cluster from my-topic . Use a terminal to run a Kafka producer and consumer on a local machine. Prerequisites Ensure AMQ Streams is installed on the OpenShift cluster. ZooKeeper and Kafka must be running to be able to send and receive messages. You need a cluster CA certificate for access to the cluster . You must be able to access to the latest version of the Red Hat AMQ Streams archive from the AMQ Streams download site . Procedure Download the latest version of the AMQ Streams archive ( amq-streams-x.y.z-bin.zip ) from the AMQ Streams download site . Unzip the file to any destination. Open a terminal, and start the Kafka console producer with the topic my-topic and the authentication properties for TLS: bin/kafka-console-producer.sh --broker-list ROUTE-ADDRESS :443 --producer-property security.protocol=SSL --producer-property ssl.truststore.password=password --producer-property ssl.truststore.location=./client.truststore.jks --topic my-topic Type your message into the console where the producer is running. Press Enter to send the message. Open a new terminal tab or window, and start the Kafka console consumer to receive the messages: bin/kafka-console-consumer.sh --bootstrap-server ROUTE-ADDRESS :443 --consumer-property security.protocol=SSL --consumer-property ssl.truststore.password=password --consumer-property ssl.truststore.location=./client.truststore.jks --topic my-topic --from-beginning Confirm that you see the incoming messages in the consumer console. Press Crtl+C to exit the Kafka console producer and consumer.
unzip amq-streams-x.y.z-ocp-install-examples.zip login -u system:admin new-project kafka sed -i 's/namespace: .*/namespace: kafka/' install/cluster-operator/*RoleBinding*.yaml sed -i '' 's/namespace: .*/namespace: kafka/' install/cluster-operator/*RoleBinding*.yaml project kafka apply -f install/cluster-operator/ new-project my-kafka-project adm policy add-role-to-user admin developer -n my-kafka-project set env deploy/strimzi-cluster-operator STRIMZI_NAMESPACE=kafka,my-kafka-project -n kafka apply -f install/cluster-operator/020-RoleBinding-strimzi-cluster-operator.yaml -n my-kafka-project apply -f install/cluster-operator/032-RoleBinding-strimzi-cluster-operator-topic-operator-delegation.yaml -n my-kafka-project apply -f install/cluster-operator/031-RoleBinding-strimzi-cluster-operator-entity-operator-delegation.yaml -n my-kafka-project apply -f install/strimzi-admin adm policy add-cluster-role-to-user strimzi-admin developer login -u developer project my-kafka-project cat << EOF | oc create -f - apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: name: my-cluster spec: kafka: replicas: 3 listeners: - name: plain port: 9092 type: internal tls: false - name: tls port: 9093 type: internal tls: true - name: external port: 9094 type: route 1 tls: true storage: type: ephemeral zookeeper: replicas: 3 storage: type: ephemeral entityOperator: topicOperator: {} EOF wait my-kafka-project/my-cluster --for=condition=Ready --timeout=300s -n kafka cat << EOF | oc create -f - apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaTopic metadata: name: my-topic labels: strimzi.io/cluster: "my-cluster" spec: partitions: 3 replicas: 3 EOF get routes my-cluster-kafka-bootstrap -o=jsonpath='{.status.ingress[0].host}{"\n"}' extract secret/my-cluster-cluster-ca-cert --keys=ca.crt --to=- > ca.crt keytool -keystore client.truststore.jks -alias CARoot -import -file ca.crt bin/kafka-console-producer.sh --broker-list ROUTE-ADDRESS :443 --producer-property security.protocol=SSL --producer-property ssl.truststore.password=password --producer-property ssl.truststore.location=./client.truststore.jks --topic my-topic bin/kafka-console-consumer.sh --bootstrap-server ROUTE-ADDRESS :443 --consumer-property security.protocol=SSL --consumer-property ssl.truststore.password=password --consumer-property ssl.truststore.location=./client.truststore.jks --topic my-topic --from-beginning