title: Graceful shutdown kafka consumer
body: <p>I'm using spring-kafka 3.1.6 and hazelcast 3.7.4.</p>
  
  <p>When shutting down my application, it seems like hazelcast is shutting down before my kafka consumer.</p>
  
  <p>Thus leading to errors</p>
  
  <pre><code>nested exception is org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!; nested exception is org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
  </code></pre>
  
  <p>and</p>
  
  <pre><code>org.apache.kafka.common.errors.WakeupException: null
  at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.maybeTriggerWakeup(ConsumerNetworkClient.java:422)
  at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:245)
  at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:208)
  </code></pre>
  
  <p>My consumer is configured with a RetryTemplate.</p>
  
  <pre><code>org.springframework.context.support.DefaultLifecycleProcessor   Failed to shut down 1 bean with phase value 2147483547 within timeout of 30000: [org.springframework.kafka.config.internalKafkaListenerEndpointRegistry]
  </code></pre>
  
  <p>Can someone help me to graceful shutdown by making hazelcast waiting for the kafka container to stop ?</p>
  
  <p>Thanks</p>
  
  <p><strong>Edit</strong></p>
  
  <p>Both KafkaContainer and HazelcastInstance are beans handled by spring.</p>
  
  <p>HazelcastInstance is created manually (not using default springboot autoconfiguration) :</p>
  
  <pre><code>@PreDestroy
  public void destroy() {
      LOGGER.info("Closing Hazelcast");
      Hazelcast.shutdownAll();
  }
  
  @Bean
  public HazelcastInstance hazelcastInstance() {
      LOGGER.debug("Configuring Hazelcast");
      Config config = new Config();
      ...
  </code></pre>
  
  <p>And Kafka Container :</p>
  
  <pre><code>@Bean(name = "kafkaListenerContainerFactory")
  public ConcurrentKafkaListenerContainerFactory&lt;Key, Value&gt; kafkaListenerContainerFactory() {
      ConcurrentKafkaListenerContainerFactory&lt;Key, Value&gt; factory =
              new ConcurrentKafkaListenerContainerFactory&lt;&gt;();
      factory.setConsumerFactory(new DefaultKafkaConsumerFactory&lt;&gt;(consumerConfigs()));
      factory.setRetryTemplate(retryTemplate());
      factory.getContainerProperties().setAckOnError(false);
      factory.getContainerProperties().setAckMode(AbstractMessageListenerContainer.AckMode.RECORD);
      factory.getContainerProperties().setErrorHandler(new DefaultErrorHandler());
      return factory;
  }
  </code></pre>
  
  <p><strong>Edit 2:</strong></p>
  
  <p>I'm running the application replicated 3 times. All 3 replicas has been shutdown at the same moment.</p>
  
  <p>Here is the shutdown logs history for 1 replica :</p>
  
  <p>1) 18:14:54.427 "ContextClosedEvent" spring event has been catched by a component and logged</p>
  
  <p>2) 18:14:54.441 org.apache.kafka.common.errors.WakeupException: null</p>
  
  <p>3) 18:14:55.951 com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!</p>
  
  <p>4) Repeat 3) 19 times from 18:14:55.951 to 18:15:23.103 </p>
  
  <p>5) 18:15:24.441 Failed to shut down 1 bean with phase value 2147483547 within timeout of 30000: [org.springframework.kafka.config.internalKafkaListenerEndpointRegistry]</p>
  
  <p>6) 18:15:24.537 Closing Hazelcast</p>