title: Request timed out Message while starting Kafka connect in distributed mode
body: <p>I have downloaded confluent version 5.4 .</p>
  
  <p><strong>And i have connector running good in stand alone mode .</strong></p>
  
  <pre><code>wget /usr/local http://packages.confluent.io/archive/5.4/confluent-5.4.0-2.11.tar.gz -P ~/Downloads/
  tar -zxvf ~/Downloads/confluent-5.4.0-2.11.tar.gz -C ~/Downloads/
  sudo mv ~/Downloads/confluent-5.4.0 /usr/local/
  </code></pre>
  
  <p>and then i am trying to run Kafak connect on distributed mode so i modified my connect-distributed.properties like below </p>
  
  <pre><code>bootstrap.servers=b-***.eu-west-1.amazonaws.com:9092,b-***.eu-west-1.amazonaws.com:9092,b-***.eu-west-1.amazonaws.com:9092
  group.id=connect-cluster
  key.converter=org.apache.kafka.connect.storage.StringConverter
  value.converter=org.apache.kafka.connect.storage.StringConverter
  
  offset.storage.topic=connect-offsets
  offset.storage.replication.factor=1
  config.storage.topic=connect-configs
  config.storage.replication.factor=1
  status.storage.topic=connect-status
  status.storage.replication.factor=1
  offset.flush.interval.ms=10000
  plugin.path=/usr/local/confluent/share/java
  </code></pre>
  
  <p>and then i start my connector like below </p>
  
  <pre><code>/usr/local/confluent/bin/connect-distributed /usr/local/confluent/etc/kafka/connect-distributed.properties
  </code></pre>
  
  <p>it looks like it started successfully [2020-02-02 05:22:33,860] INFO Joined group and got assignment: </p>
  
  <pre><code>Assignment{error=0, leader='connect-1-c99d50a9-faf0-4b15-8a3d-3add55b7e206', leaderUrl='http://10.97.49.217:8083/', offset=-1, connectorIds=[], taskIds=[]} (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1216)
  [2020-02-02 05:22:33,861] INFO Starting connectors and tasks using config offset -1 (org.apache.kafka.connect.runtime.distributed.DistributedHerder:850)
  [2020-02-02 05:22:33,861] INFO Finished starting connectors and tasks (org.apache.kafka.connect.runtime.distributed.DistributedHerder:860)
  </code></pre>
  
  <p>Finally i am using curl command to create my sink connector .</p>
  
  <pre><code>curl  -X POST -H "Content-Type: application/json" --data '{"name":"elastic-search-sink-audit","config":{"connector.class":"io.confluent.connect.elasticsearch.ElasticsearchSinkConnector","tasks.max":"2","topics":"fsp-AUDIT_EVENT_DEMO","key.ignore":"true","connection.url":"https://**.amazonaws.com","type.name":"kafka-connect","name":"elastic-search-sink-audit","errors.tolerance":"all","errors.deadletterqueue.topic.name":"fsp-dlq-audit-event"}}' http://localhost:8083/connectors | jq
  </code></pre>
  
  <p>and with EC2 IP address </p>
  
  <pre><code>curl  -X POST -H "Content-Type: application/json" --data '{"name":"elastic-search-sink-audit-distributed","config":{"connector.class":"io.confluent.connect.elasticsearch.ElasticsearchSinkConnector","tasks.max":"2","topics":"audit_event_distributed","key.ignore":"true","connection.url":"https://***.us-east-1.es.amazonaws.com","type.name":"kafka-connect","name":"elastic-search-sink-audit-distributed","errors.tolerance":"all","errors.deadletterqueue.topic.name":"dlq_distributed"}}' http://10.67.39.217:8083/connectors | jq
  </code></pre>
  
  <p>After running this command after 1 minute i get below response </p>
  
  <pre><code>  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed
  100   498    0    48  100   450      0      4  0:01:52  0:01:30  0:00:22    10
  {
    "error_code": 500,
    "message": "Request timed out"
  }
  </code></pre>
  
  <p>And then in i keep on getting this as well but this is just warning as i have enough replicas in my topic </p>
  
  <pre><code>NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender:598)
  ^C[2020-02-03 08:05:54,890] WARN [Producer clientId=producer-3] Got error produce response with correlation id 7185 on topic-partition connect-configs-0, retrying (2147476495 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender:598)
  ^C[2020-02-03 08:05:54,991] WARN [Producer clientId=producer-3] Got error produce response with correlation id 7186 on topic-partition connect-configs-0, retrying (2147476494 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender:598)
  </code></pre>
  
  <p>When i described my topic </p>
  
  <pre><code>Topic:fsp-AUDIT_EVENT_DEMO      PartitionCount:1        ReplicationFactor:3     Configs:
          Topic: fsp-AUDIT_EVENT_DEMO     Partition: 0    Leader: 1       Replicas: 1,6,2 Isr: 1,6,2
  </code></pre>
  
  <p>So i have enough replica for my topic .</p>
  
  <p><strong>UPDATE</strong></p>
  
  <p>Describe result of </p>
  
  <pre><code>Topic:connect-configs   PartitionCount:1        ReplicationFactor:1     Configs:cleanup.policy=compact
          Topic: connect-configs  Partition: 0    Leader: 6       Replicas: 6     Isr: 6
  
  
  Topic:connect-status    PartitionCount:5        ReplicationFactor:1     Configs:cleanup.policy=compact
          Topic: connect-status   Partition: 0    Leader: 6       Replicas: 6     Isr: 6
          Topic: connect-status   Partition: 1    Leader: 1       Replicas: 1     Isr: 1
          Topic: connect-status   Partition: 2    Leader: 2       Replicas: 2     Isr: 2
          Topic: connect-status   Partition: 3    Leader: 4       Replicas: 4     Isr: 4
          Topic: connect-status   Partition: 4    Leader: 5       Replicas: 5     Isr: 5
  
  
  Topic:connect-offsets   PartitionCount:25       ReplicationFactor:1     Configs:cleanup.policy=compact
  </code></pre>
  
  <p>Please help me resolving this .</p>