title: Docker compose create kafka topics
body: <p><strong>Problem</strong>: Cannot create topics from docker-compose. I need to create kafka topics before I run a system under test. Planning to use it as a part of the pipeline, hence using UI is not an option.</p>
  <p>Note: it takes ~15 seconds for kafka to be ready so I would need to put a sleep for 15 seconds prior to adding the topics.</p>
  <p><strong>Possible solution</strong>:</p>
  <ol>
  <li>create a shell.sh file with commands to wait for 15 sec then add a bunch of topics</li>
  <li>create a dockerfile for it</li>
  <li>include that docker image in the docker-compose.yml just before starting the system under test</li>
  </ol>
  <p><strong>Current flow</strong>:</p>
  <ol>
  <li>create zookeeper - OK</li>
  <li>create kafka1 - OK</li>
  <li>rest-proxy - OK</li>
  <li>create topics <strong>&lt;- PROBLEM</strong></li>
  <li>create SUT - OK</li>
  </ol>
  <p><strong>Current docker-compose.yml</strong>:</p>
  <pre><code>version: '2'
  services:
  zookeeper:
  image: docker.io/confluentinc/cp-zookeeper:5.4.1
  hostname: zookeeper
  container_name: zookeeper
  ports:
  - &quot;2181:2181&quot;
  environment:
  ZOOKEEPER_CLIENT_PORT: 2181
  ZOOKEEPER_TICK_TIME: 2000
  
  Kafka1:
  image: docker.io/confluentinc/cp-enterprise-kafka:5.4.1
  hostname: Kafka1
  container_name: Kafka1
  depends_on:
  - zookeeper
  ports:
  - &quot;9092:9092&quot;
  - &quot;9101:9101&quot;
  environment:
  KAFKA_BROKER_ID: 1
  KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
  KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
  KAFKA_ADVERTISED_HOST_NAME: Kafka1
  KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://Kafka1:29092,PLAINTEXT_HOST://localhost:9092
  KAFKA_METRIC_REPORTERS: io.confluent.metrics.reporter.ConfluentMetricsReporter
  KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
  KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
  KAFKA_CONFLUENT_LICENSE_TOPIC_REPLICATION_FACTOR: 1
  KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
  KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
  KAFKA_JMX_PORT: 9101
  KAFKA_CONFLUENT_SCHEMA_REGISTRY_URL: http://schema-registry:8081
  CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: Kafka1:29092
  CONFLUENT_METRICS_REPORTER_ZOOKEEPER_CONNECT: zookeeper:2181
  CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 1
  CONFLUENT_METRICS_ENABLE: 'true'
  CONFLUENT_SUPPORT_CUSTOMER_ID: 'anonymous'
  
  rest-proxy:
  image: docker.io/confluentinc/cp-kafka-rest:5.4.1
  depends_on:
  - zookeeper
  - Kafka1
  ports:
  - 8082:8082
  hostname: rest-proxy
  container_name: rest-proxy
  environment:
  KAFKA_REST_HOST_NAME: rest-proxy
  KAFKA_REST_BOOTSTRAP_SERVERS: 'Kafka1:29092'
  KAFKA_REST_LISTENERS: &quot;http://0.0.0.0:8082&quot;
  KAFKA_REST_SCHEMA_REGISTRY_URL: 'http://schema-registry:8081'
  
  
  topics:
  image: topics:latest
  hostname: topics
  container_name: topics
  depends_on:
  - zookeeper
  - Kafka1
  - rest-proxy
  
  
  sut:
  image: sut:latest
  hostname: sut
  container_name: sut
  depends_on:
  - zookeeper
  - Kafka1
  - rest-proxy
  ports:
  - 5000:80
  </code></pre>
  <p><strong>Current Dockerfile for topics container</strong>:</p>
  <pre><code>FROM ubuntu:14.04
  
  ADD topics.sh /usr/local/bin/topics.sh
  
  RUN chmod +x /usr/local/bin/topics.sh
  
  CMD /usr/local/bin/topics.sh
  </code></pre>
  <p><strong>Current topics.sh file</strong>:</p>
  <pre><code>#!/bin/sh
  echo &quot;Start: Sleep 15 seconds&quot;
  sleep 30;
  wait;
  echo &quot;Begin creating topics&quot;
  docker exec Kafka1 kafka-topics --create --if-not-exists --zookeeper zookeeper:2181 --partitions 1 --replication-factor 1 --topic MY_AWESOME_TOPIC_ONE
  docker exec Kafka1 kafka-topics --create --if-not-exists --zookeeper zookeeper:2181 --partitions 1 --replication-factor 1 --topic MY_AWESOME_TOPIC_TWO
  echo &quot;Done creating topics&quot;
  </code></pre>
  <p><strong>Current output</strong>:</p>
  <pre><code>/usr/local/bin/topics.sh: 1: /usr/local/bin/topics.sh: ï»¿#!/bin/sh: not found
  Start: Sleep 15 seconds
  Begin creating topics
  /usr/local/bin/topics.sh: 8: /usr/local/bin/topics.sh: docker: not found
  /usr/local/bin/topics.sh: 9: /usr/local/bin/topics.sh: docker: not found
  /usr/local/bin/topics.sh: 10: /usr/local/bin/topics.sh: docker: not found
  /usr/local/bin/topics.sh: 11: /usr/local/bin/topics.sh: docker: not found
  /usr/local/bin/topics.sh: 12: /usr/local/bin/topics.sh: docker: not found
  /usr/local/bin/topics.sh: 13: /usr/local/bin/topics.sh: docker: not found
  /usr/local/bin/topics.sh: 14: /usr/local/bin/topics.sh: docker: not found
  /usr/local/bin/topics.sh: 15: /usr/local/bin/topics.sh: docker: not found
  Done creating topics
  </code></pre>
  <p>Topics are not created. I'm stuck. Please help.</p>