title: ActiveMQ Artemis cluster does not redistribute messages after one instance crash
body: <p>I have a cluster of Artemis in Kubernetes with 3 group of master/slave:</p>
  <pre><code>activemq-artemis-master-0                               1/1     Running
  activemq-artemis-master-1                               1/1     Running
  activemq-artemis-master-2                               1/1     Running
  activemq-artemis-slave-0                                0/1     Running
  activemq-artemis-slave-1                                0/1     Running
  activemq-artemis-slave-2                                0/1     Running
  </code></pre>
  <p>I am using Spring boot JmsListener to consume messages sent to a wildcard queue as follow.</p>
  <pre class="lang-java prettyprint-override"><code>    @Component
      @Log4j2
      public class QueueListener {
        @Autowired
        private ListenerControl listenerControl;
      
        @JmsListener(id = &quot;queueListener0&quot;, destination = &quot;QUEUE.service2.*.*.*.notification&quot;)
        public void add(String message, @Header(&quot;sentBy&quot;) String sentBy, @Header(&quot;sentFrom&quot;) String sentFrom, @Header(&quot;sentAt&quot;) Long sentAt) throws InterruptedException {
      
      
          log.info(&quot;---QUEUE[notification]:  message={}, sentBy={}, sentFrom={}, sentAt={}&quot;,
                      message, sentBy, sentFrom, sentAt);
      
          TimeUnit.MILLISECONDS.sleep(listenerControl.getDuration());
        }
      }
  </code></pre>
  <p>There was 20 messages sent to the queue and master-1 was the delivering node. When 5 messages has been consumed, I killed the master-1 node to simulate a crash, I saw slave-1 started running then yielded back to master-1 after Kubernetes respawn it. The listener threw a <code>JMSException</code> that the connection was lost and it tried to reconnect. Then I saw it successfully connected to master-0 (I saw the queue created and the consumer count &gt; 0). However the queue on master-0 was empty, while the same queue in master-1 still had 15 messages and no consumer attached to it. I waited for a while but the 15 messages was never delivered. I am not sure why redistribution did not kick in.</p>
  <p>The attributes of the wildcard queue on master-1 is like this when it came back online after the crash (I manually replace the value of the field <strong>accessToken</strong> since it has sensitive info):</p>
  <pre><code>Attribute   Value
  Acknowledge attempts    0
  Address QUEUE.service2.*.*.*.notification
  Configuration managed   false
  Consumer count  0
  Consumers before dispatch   0
  Dead letter address DLQ
  Delay before dispatch   -1
  Delivering count    0
  Delivering size 0
  Durable true
  Durable delivering count    0
  Durable delivering size 0
  Durable message count   15
  Durable persistent size 47705
  Durable scheduled count 0
  Durable scheduled size  0
  Enabled true
  Exclusive   false
  Expiry address  ExpiryQueue
  Filter  
  First message age   523996
  First message as json   [{&quot;JMSType&quot;:&quot;service2&quot;,&quot;address&quot;:&quot;QUEUE.service2.tech-drive2.188100000059.thai.notification&quot;,&quot;messageID&quot;:68026,&quot;sentAt&quot;:1621957145988,&quot;accessToken&quot;:&quot;REMOVED&quot;,&quot;type&quot;:3,&quot;priority&quot;:4,&quot;userID&quot;:&quot;ID:56c7b509-bd6f-11eb-a348-de0dacf99072&quot;,&quot;_AMQ_GROUP_ID&quot;:&quot;tech-drive2-188100000059-thai&quot;,&quot;sentBy&quot;:&quot;user@email.com&quot;,&quot;durable&quot;:true,&quot;JMSReplyTo&quot;:&quot;queue://QUEUE.service2.tech-drive2.188100000059.thai.notification&quot;,&quot;__AMQ_CID&quot;:&quot;e4469ea3-bd62-11eb-a348-de0dacf99072&quot;,&quot;sentFrom&quot;:&quot;service2&quot;,&quot;originalDestination&quot;:&quot;QUEUE.service2.tech-drive2.188100000059.thai.notification&quot;,&quot;_AMQ_ROUTING_TYPE&quot;:1,&quot;JMSCorrelationID&quot;:&quot;c329c733-1170-440a-9080-992a009d87a9&quot;,&quot;expiration&quot;:0,&quot;timestamp&quot;:1621957145988}]
  First message timestamp 1621957145988
  Group buckets   -1
  Group count 0
  Group first key 
  Group rebalance false
  Group rebalance pause dispatch  false
  Id  119
  Last value  false
  Last value key  
  Max consumers   -1
  Message count   15
  Messages acknowledged   0
  Messages added  15
  Messages expired    0
  Messages killed 0
  Name    QUEUE.service2.*.*.*.notification
  Object Name org.apache.activemq.artemis:broker=&quot;activemq-artemis-master-1&quot;,component=addresses,address=&quot;QUEUE.service2.\*.\*.\*.notification&quot;,subcomponent=queues,routing-type=&quot;anycast&quot;,queue=&quot;QUEUE.service2.\*.\*.\*.notification&quot;
  Paused  false
  Persistent size 47705
  Prepared transaction message count  0
  Purge on no consumers   false
  Retroactive resource    false
  Ring size   -1
  Routing type    ANYCAST
  Scheduled count 0
  Scheduled size  0
  Temporary   false
  User    f7bcdaed-8c0c-4bb5-ad03-ec06382cb557
  </code></pre>
  <p>The attributes of the wildcard queue on master-0 is like this:</p>
  <pre><code>Attribute   Value
  Acknowledge attempts    0
  Address QUEUE.service2.*.*.*.notification
  Configuration managed   false
  Consumer count  3
  Consumers before dispatch   0
  Dead letter address DLQ
  Delay before dispatch   -1
  Delivering count    0
  Delivering size 0
  Durable true
  Durable delivering count    0
  Durable delivering size 0
  Durable message count   0
  Durable persistent size 0
  Durable scheduled count 0
  Durable scheduled size  0
  Enabled true
  Exclusive   false
  Expiry address  ExpiryQueue
  Filter  
  First message age   
  First message as json   [{}]
  First message timestamp 
  Group buckets   -1
  Group count 0
  Group first key 
  Group rebalance false
  Group rebalance pause dispatch  false
  Id  119
  Last value  false
  Last value key  
  Max consumers   -1
  Message count   0
  Messages acknowledged   0
  Messages added  0
  Messages expired    0
  Messages killed 0
  Name    QUEUE.service2.*.*.*.notification
  Object Name org.apache.activemq.artemis:broker=&quot;activemq-artemis-master-0&quot;,component=addresses,address=&quot;QUEUE.service2.\*.\*.\*.notification&quot;,subcomponent=queues,routing-type=&quot;anycast&quot;,queue=&quot;QUEUE.service2.\*.\*.\*.notification&quot;
  Paused  false
  Persistent size 0
  Prepared transaction message count  0
  Purge on no consumers   false
  Retroactive resource    false
  Ring size   -1
  Routing type    ANYCAST
  Scheduled count 0
  Scheduled size  0
  Temporary   false
  User    f7bcdaed-8c0c-4bb5-ad03-ec06382cb557
  </code></pre>
  <p>The Artemis version in use is 2.17.0. Here is my cluster config in master-0 <code>broker.xml</code>. The configs are the same for other brokers except the <code>connector-ref</code> is changed to match the broker:</p>
  <pre class="lang-xml prettyprint-override"><code>&lt;?xml version=&quot;1.0&quot;?&gt;
  &lt;configuration xmlns=&quot;urn:activemq&quot; xmlns:xi=&quot;http://www.w3.org/2001/XInclude&quot; xmlns:xsi=&quot;http://www.w3.org/2001/XMLSchema-instance&quot; xsi:schemaLocation=&quot;urn:activemq /schema/artemis-configuration.xsd&quot;&gt;
    &lt;core xmlns=&quot;urn:activemq:core&quot; xsi:schemaLocation=&quot;urn:activemq:core &quot;&gt;
      &lt;name&gt;activemq-artemis-master-0&lt;/name&gt;
      &lt;persistence-enabled&gt;true&lt;/persistence-enabled&gt;
      &lt;journal-type&gt;ASYNCIO&lt;/journal-type&gt;
      &lt;paging-directory&gt;data/paging&lt;/paging-directory&gt;
      &lt;bindings-directory&gt;data/bindings&lt;/bindings-directory&gt;
      &lt;journal-directory&gt;data/journal&lt;/journal-directory&gt;
      &lt;large-messages-directory&gt;data/large-messages&lt;/large-messages-directory&gt;
      &lt;journal-datasync&gt;true&lt;/journal-datasync&gt;
      &lt;journal-min-files&gt;2&lt;/journal-min-files&gt;
      &lt;journal-pool-files&gt;10&lt;/journal-pool-files&gt;
      &lt;journal-device-block-size&gt;4096&lt;/journal-device-block-size&gt;
      &lt;journal-file-size&gt;10M&lt;/journal-file-size&gt;
      &lt;journal-buffer-timeout&gt;100000&lt;/journal-buffer-timeout&gt;
      &lt;journal-max-io&gt;4096&lt;/journal-max-io&gt;
      &lt;disk-scan-period&gt;5000&lt;/disk-scan-period&gt;
      &lt;max-disk-usage&gt;90&lt;/max-disk-usage&gt;
      &lt;critical-analyzer&gt;true&lt;/critical-analyzer&gt;
      &lt;critical-analyzer-timeout&gt;120000&lt;/critical-analyzer-timeout&gt;
      &lt;critical-analyzer-check-period&gt;60000&lt;/critical-analyzer-check-period&gt;
      &lt;critical-analyzer-policy&gt;HALT&lt;/critical-analyzer-policy&gt;
      &lt;page-sync-timeout&gt;2244000&lt;/page-sync-timeout&gt;
      &lt;acceptors&gt;
        &lt;acceptor name=&quot;artemis&quot;&gt;tcp://0.0.0.0:61616?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;amqpMinLargeMessageSize=102400;protocols=CORE,AMQP,STOMP,HORNETQ,MQTT,OPENWIRE;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpDuplicateDetection=true&lt;/acceptor&gt;
        &lt;acceptor name=&quot;amqp&quot;&gt;tcp://0.0.0.0:5672?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=AMQP;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpMinLargeMessageSize=102400;amqpDuplicateDetection=true&lt;/acceptor&gt;
        &lt;acceptor name=&quot;stomp&quot;&gt;tcp://0.0.0.0:61613?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=STOMP;useEpoll=true&lt;/acceptor&gt;
        &lt;acceptor name=&quot;hornetq&quot;&gt;tcp://0.0.0.0:5445?anycastPrefix=jms.queue.;multicastPrefix=jms.topic.;protocols=HORNETQ,STOMP;useEpoll=true&lt;/acceptor&gt;
        &lt;acceptor name=&quot;mqtt&quot;&gt;tcp://0.0.0.0:1883?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=MQTT;useEpoll=true&lt;/acceptor&gt;
      &lt;/acceptors&gt;
      &lt;security-settings&gt;
        &lt;security-setting match=&quot;#&quot;&gt;
          &lt;permission type=&quot;createNonDurableQueue&quot; roles=&quot;amq&quot;/&gt;
          &lt;permission type=&quot;deleteNonDurableQueue&quot; roles=&quot;amq&quot;/&gt;
          &lt;permission type=&quot;createDurableQueue&quot; roles=&quot;amq&quot;/&gt;
          &lt;permission type=&quot;deleteDurableQueue&quot; roles=&quot;amq&quot;/&gt;
          &lt;permission type=&quot;createAddress&quot; roles=&quot;amq&quot;/&gt;
          &lt;permission type=&quot;deleteAddress&quot; roles=&quot;amq&quot;/&gt;
          &lt;permission type=&quot;consume&quot; roles=&quot;amq&quot;/&gt;
          &lt;permission type=&quot;browse&quot; roles=&quot;amq&quot;/&gt;
          &lt;permission type=&quot;send&quot; roles=&quot;amq&quot;/&gt;
          &lt;permission type=&quot;manage&quot; roles=&quot;amq&quot;/&gt;
        &lt;/security-setting&gt;
      &lt;/security-settings&gt;
      &lt;address-settings&gt;
        &lt;address-setting match=&quot;activemq.management#&quot;&gt;
          &lt;dead-letter-address&gt;DLQ&lt;/dead-letter-address&gt;
          &lt;expiry-address&gt;ExpiryQueue&lt;/expiry-address&gt;
          &lt;redelivery-delay&gt;0&lt;/redelivery-delay&gt;
          &lt;!-- with -1 only the global-max-size is in use for limiting --&gt;
          &lt;max-size-bytes&gt;-1&lt;/max-size-bytes&gt;
          &lt;message-counter-history-day-limit&gt;10&lt;/message-counter-history-day-limit&gt;
          &lt;address-full-policy&gt;PAGE&lt;/address-full-policy&gt;
          &lt;auto-create-queues&gt;true&lt;/auto-create-queues&gt;
          &lt;auto-create-addresses&gt;true&lt;/auto-create-addresses&gt;
          &lt;auto-create-jms-queues&gt;true&lt;/auto-create-jms-queues&gt;
          &lt;auto-create-jms-topics&gt;true&lt;/auto-create-jms-topics&gt;
        &lt;/address-setting&gt;
        &lt;address-setting match=&quot;#&quot;&gt;
          &lt;dead-letter-address&gt;DLQ&lt;/dead-letter-address&gt;
          &lt;expiry-address&gt;ExpiryQueue&lt;/expiry-address&gt;
          &lt;redistribution-delay&gt;60000&lt;/redistribution-delay&gt;
          &lt;redelivery-delay&gt;0&lt;/redelivery-delay&gt;
          &lt;max-size-bytes&gt;-1&lt;/max-size-bytes&gt;
          &lt;message-counter-history-day-limit&gt;10&lt;/message-counter-history-day-limit&gt;
          &lt;address-full-policy&gt;PAGE&lt;/address-full-policy&gt;
          &lt;auto-create-queues&gt;true&lt;/auto-create-queues&gt;
          &lt;auto-create-addresses&gt;true&lt;/auto-create-addresses&gt;
          &lt;auto-create-jms-queues&gt;true&lt;/auto-create-jms-queues&gt;
          &lt;auto-create-jms-topics&gt;true&lt;/auto-create-jms-topics&gt;
        &lt;/address-setting&gt;
      &lt;/address-settings&gt;
      &lt;addresses&gt;
        &lt;address name=&quot;DLQ&quot;&gt;
          &lt;anycast&gt;
            &lt;queue name=&quot;DLQ&quot;/&gt;
          &lt;/anycast&gt;
        &lt;/address&gt;
        &lt;address name=&quot;ExpiryQueue&quot;&gt;
          &lt;anycast&gt;
            &lt;queue name=&quot;ExpiryQueue&quot;/&gt;
          &lt;/anycast&gt;
        &lt;/address&gt;
      &lt;/addresses&gt;
      &lt;cluster-user&gt;clusterUser&lt;/cluster-user&gt;
      &lt;cluster-password&gt;aShortclusterPassword&lt;/cluster-password&gt;
      &lt;connectors&gt;
        &lt;connector name=&quot;activemq-artemis-master-0&quot;&gt;tcp://activemq-artemis-master-0.activemq-artemis-master.svc.cluster.local:61616&lt;/connector&gt;
        &lt;connector name=&quot;activemq-artemis-slave-0&quot;&gt;tcp://activemq-artemis-slave-0.activemq-artemis-slave.svc.cluster.local:61616&lt;/connector&gt;
        &lt;connector name=&quot;activemq-artemis-master-1&quot;&gt;tcp://activemq-artemis-master-1.activemq-artemis-master.svc.cluster.local:61616&lt;/connector&gt;
        &lt;connector name=&quot;activemq-artemis-slave-1&quot;&gt;tcp://activemq-artemis-slave-1.activemq-artemis-slave.svc.cluster.local:61616&lt;/connector&gt;
        &lt;connector name=&quot;activemq-artemis-master-2&quot;&gt;tcp://activemq-artemis-master-2.activemq-artemis-master.svc.cluster.local:61616&lt;/connector&gt;
        &lt;connector name=&quot;activemq-artemis-slave-2&quot;&gt;tcp://activemq-artemis-slave-2.activemq-artemis-slave.svc.cluster.local:61616&lt;/connector&gt;
      &lt;/connectors&gt;
      &lt;cluster-connections&gt;
        &lt;cluster-connection name=&quot;activemq-artemis&quot;&gt;
          &lt;connector-ref&gt;activemq-artemis-master-0&lt;/connector-ref&gt;
          &lt;retry-interval&gt;500&lt;/retry-interval&gt;
          &lt;retry-interval-multiplier&gt;1.1&lt;/retry-interval-multiplier&gt;
          &lt;max-retry-interval&gt;5000&lt;/max-retry-interval&gt;
          &lt;initial-connect-attempts&gt;-1&lt;/initial-connect-attempts&gt;
          &lt;reconnect-attempts&gt;-1&lt;/reconnect-attempts&gt;
          &lt;message-load-balancing&gt;ON_DEMAND&lt;/message-load-balancing&gt;
          &lt;max-hops&gt;1&lt;/max-hops&gt;
          &lt;!-- scale-down&gt;true&lt;/scale-down --&gt;
          &lt;static-connectors&gt;
            &lt;connector-ref&gt;activemq-artemis-master-0&lt;/connector-ref&gt;
            &lt;connector-ref&gt;activemq-artemis-slave-0&lt;/connector-ref&gt;
            &lt;connector-ref&gt;activemq-artemis-master-1&lt;/connector-ref&gt;
            &lt;connector-ref&gt;activemq-artemis-slave-1&lt;/connector-ref&gt;
            &lt;connector-ref&gt;activemq-artemis-master-2&lt;/connector-ref&gt;
            &lt;connector-ref&gt;activemq-artemis-slave-2&lt;/connector-ref&gt;
          &lt;/static-connectors&gt;
        &lt;/cluster-connection&gt;
      &lt;/cluster-connections&gt;
      &lt;ha-policy&gt;
        &lt;replication&gt;
          &lt;master&gt;
            &lt;group-name&gt;activemq-artemis-0&lt;/group-name&gt;
            &lt;quorum-vote-wait&gt;12&lt;/quorum-vote-wait&gt;
            &lt;vote-on-replication-failure&gt;true&lt;/vote-on-replication-failure&gt;
            &lt;!--we need this for auto failback--&gt;
            &lt;check-for-live-server&gt;true&lt;/check-for-live-server&gt;
          &lt;/master&gt;
        &lt;/replication&gt;
      &lt;/ha-policy&gt;
    &lt;/core&gt;
    &lt;core xmlns=&quot;urn:activemq:core&quot;&gt;
      &lt;jmx-management-enabled&gt;true&lt;/jmx-management-enabled&gt;
    &lt;/core&gt;
  &lt;/configuration&gt;
  </code></pre>
  <p>From another answer from Stack Overflow, I understand that my topology for high-availability is redundant and I am planning to remove the slave. However, I don't think the slave is the cause for redistribution of messages not working. Is there a config that I am missing to handle Artemis node crash?</p>
  <p>Updated 1:
  As Justin suggested, I tried to use a cluster of 2 nodes of Artemis without HA.</p>
  <pre><code>activemq-artemis-master-0                              1/1     Running            0          27m
  activemq-artemis-master-1                              1/1     Running            0          74s
  </code></pre>
  <p>The following is broker.xml of the 2 artemis node. The only different between them is the node name and journal-buffer-timeout:</p>
  <pre><code>&lt;?xml version=&quot;1.0&quot;?&gt;
  
  &lt;configuration xmlns=&quot;urn:activemq&quot; xmlns:xi=&quot;http://www.w3.org/2001/XInclude&quot; xmlns:xsi=&quot;http://www.w3.org/2001/XMLSchema-instance&quot; xsi:schemaLocation=&quot;urn:activemq /schema/artemis-configuration.xsd&quot;&gt;
    &lt;core xmlns=&quot;urn:activemq:core&quot; xsi:schemaLocation=&quot;urn:activemq:core &quot;&gt;
      &lt;name&gt;activemq-artemis-master-0&lt;/name&gt;
      &lt;persistence-enabled&gt;true&lt;/persistence-enabled&gt;
      &lt;journal-type&gt;ASYNCIO&lt;/journal-type&gt;
      &lt;paging-directory&gt;data/paging&lt;/paging-directory&gt;
      &lt;bindings-directory&gt;data/bindings&lt;/bindings-directory&gt;
      &lt;journal-directory&gt;data/journal&lt;/journal-directory&gt;
      &lt;large-messages-directory&gt;data/large-messages&lt;/large-messages-directory&gt;
      &lt;journal-datasync&gt;true&lt;/journal-datasync&gt;
      &lt;journal-min-files&gt;2&lt;/journal-min-files&gt;
      &lt;journal-pool-files&gt;10&lt;/journal-pool-files&gt;
      &lt;journal-device-block-size&gt;4096&lt;/journal-device-block-size&gt;
      &lt;journal-file-size&gt;10M&lt;/journal-file-size&gt;
      &lt;journal-buffer-timeout&gt;100000&lt;/journal-buffer-timeout&gt;
      &lt;journal-max-io&gt;4096&lt;/journal-max-io&gt;
      &lt;disk-scan-period&gt;5000&lt;/disk-scan-period&gt;
      &lt;max-disk-usage&gt;90&lt;/max-disk-usage&gt;
      &lt;critical-analyzer&gt;true&lt;/critical-analyzer&gt;
      &lt;critical-analyzer-timeout&gt;120000&lt;/critical-analyzer-timeout&gt;
      &lt;critical-analyzer-check-period&gt;60000&lt;/critical-analyzer-check-period&gt;
      &lt;critical-analyzer-policy&gt;HALT&lt;/critical-analyzer-policy&gt;
      &lt;page-sync-timeout&gt;2244000&lt;/page-sync-timeout&gt;
      &lt;acceptors&gt;
        &lt;acceptor name=&quot;artemis&quot;&gt;tcp://0.0.0.0:61616?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;amqpMinLargeMessageSize=102400;protocols=CORE,AMQP,STOMP,HORNETQ,MQTT,OPENWIRE;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpDuplicateDetection=true&lt;/acceptor&gt;
        &lt;acceptor name=&quot;amqp&quot;&gt;tcp://0.0.0.0:5672?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=AMQP;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpMinLargeMessageSize=102400;amqpDuplicateDetection=true&lt;/acceptor&gt;
        &lt;acceptor name=&quot;stomp&quot;&gt;tcp://0.0.0.0:61613?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=STOMP;useEpoll=true&lt;/acceptor&gt;
        &lt;acceptor name=&quot;hornetq&quot;&gt;tcp://0.0.0.0:5445?anycastPrefix=jms.queue.;multicastPrefix=jms.topic.;protocols=HORNETQ,STOMP;useEpoll=true&lt;/acceptor&gt;
        &lt;acceptor name=&quot;mqtt&quot;&gt;tcp://0.0.0.0:1883?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=MQTT;useEpoll=true&lt;/acceptor&gt;
      &lt;/acceptors&gt;
      &lt;security-settings&gt;
        &lt;security-setting match=&quot;#&quot;&gt;
          &lt;permission type=&quot;createNonDurableQueue&quot; roles=&quot;amq&quot;/&gt;
          &lt;permission type=&quot;deleteNonDurableQueue&quot; roles=&quot;amq&quot;/&gt;
          &lt;permission type=&quot;createDurableQueue&quot; roles=&quot;amq&quot;/&gt;
          &lt;permission type=&quot;deleteDurableQueue&quot; roles=&quot;amq&quot;/&gt;
          &lt;permission type=&quot;createAddress&quot; roles=&quot;amq&quot;/&gt;
          &lt;permission type=&quot;deleteAddress&quot; roles=&quot;amq&quot;/&gt;
          &lt;permission type=&quot;consume&quot; roles=&quot;amq&quot;/&gt;
          &lt;permission type=&quot;browse&quot; roles=&quot;amq&quot;/&gt;
          &lt;permission type=&quot;send&quot; roles=&quot;amq&quot;/&gt;
          &lt;permission type=&quot;manage&quot; roles=&quot;amq&quot;/&gt;
        &lt;/security-setting&gt;
      &lt;/security-settings&gt;
      &lt;cluster-user&gt;ClusterUser&lt;/cluster-user&gt;
      &lt;cluster-password&gt;longClusterPassword&lt;/cluster-password&gt;
      &lt;connectors&gt;
        &lt;connector name=&quot;activemq-artemis-master-0&quot;&gt;tcp://activemq-artemis-master-0.activemq-artemis-master.ncp-stack-testing.svc.cluster.local:61616&lt;/connector&gt;
        &lt;connector name=&quot;activemq-artemis-master-1&quot;&gt;tcp://activemq-artemis-master-1.activemq-artemis-master.ncp-stack-testing.svc.cluster.local:61616&lt;/connector&gt;
      &lt;/connectors&gt;
      &lt;cluster-connections&gt;
        &lt;cluster-connection name=&quot;activemq-artemis&quot;&gt;
          &lt;connector-ref&gt;activemq-artemis-master-0&lt;/connector-ref&gt;
          &lt;retry-interval&gt;500&lt;/retry-interval&gt;
          &lt;retry-interval-multiplier&gt;1.1&lt;/retry-interval-multiplier&gt;
          &lt;max-retry-interval&gt;5000&lt;/max-retry-interval&gt;
          &lt;initial-connect-attempts&gt;-1&lt;/initial-connect-attempts&gt;
          &lt;reconnect-attempts&gt;-1&lt;/reconnect-attempts&gt;
          &lt;use-duplicate-detection&gt;true&lt;/use-duplicate-detection&gt;
          &lt;message-load-balancing&gt;ON_DEMAND&lt;/message-load-balancing&gt;
          &lt;max-hops&gt;1&lt;/max-hops&gt;
          &lt;static-connectors&gt;
            &lt;connector-ref&gt;activemq-artemis-master-0&lt;/connector-ref&gt;
            &lt;connector-ref&gt;activemq-artemis-master-1&lt;/connector-ref&gt;
          &lt;/static-connectors&gt;
        &lt;/cluster-connection&gt;
      &lt;/cluster-connections&gt;
      &lt;address-settings&gt;
        &lt;address-setting match=&quot;activemq.management#&quot;&gt;
          &lt;dead-letter-address&gt;DLQ&lt;/dead-letter-address&gt;
          &lt;expiry-address&gt;ExpiryQueue&lt;/expiry-address&gt;
          &lt;redelivery-delay&gt;0&lt;/redelivery-delay&gt;
          &lt;max-size-bytes&gt;-1&lt;/max-size-bytes&gt;
          &lt;message-counter-history-day-limit&gt;10&lt;/message-counter-history-day-limit&gt;
          &lt;address-full-policy&gt;PAGE&lt;/address-full-policy&gt;
          &lt;auto-create-queues&gt;true&lt;/auto-create-queues&gt;
          &lt;auto-create-addresses&gt;true&lt;/auto-create-addresses&gt;
          &lt;auto-create-jms-queues&gt;true&lt;/auto-create-jms-queues&gt;
          &lt;auto-create-jms-topics&gt;true&lt;/auto-create-jms-topics&gt;
        &lt;/address-setting&gt;
        &lt;address-setting match=&quot;#&quot;&gt;
          &lt;dead-letter-address&gt;DLQ&lt;/dead-letter-address&gt;
          &lt;expiry-address&gt;ExpiryQueue&lt;/expiry-address&gt;
          &lt;redistribution-delay&gt;60000&lt;/redistribution-delay&gt;
          &lt;redelivery-delay&gt;0&lt;/redelivery-delay&gt;
          &lt;max-size-bytes&gt;-1&lt;/max-size-bytes&gt;
          &lt;message-counter-history-day-limit&gt;10&lt;/message-counter-history-day-limit&gt;
          &lt;address-full-policy&gt;PAGE&lt;/address-full-policy&gt;
          &lt;auto-create-queues&gt;true&lt;/auto-create-queues&gt;
          &lt;auto-create-addresses&gt;true&lt;/auto-create-addresses&gt;
          &lt;auto-create-jms-queues&gt;true&lt;/auto-create-jms-queues&gt;
          &lt;auto-create-jms-topics&gt;true&lt;/auto-create-jms-topics&gt;
        &lt;/address-setting&gt;
      &lt;/address-settings&gt;
      &lt;addresses&gt;
        &lt;address name=&quot;DLQ&quot;&gt;
          &lt;anycast&gt;
            &lt;queue name=&quot;DLQ&quot;/&gt;
          &lt;/anycast&gt;
        &lt;/address&gt;
        &lt;address name=&quot;ExpiryQueue&quot;&gt;
          &lt;anycast&gt;
            &lt;queue name=&quot;ExpiryQueue&quot;/&gt;
          &lt;/anycast&gt;
        &lt;/address&gt;
      &lt;/addresses&gt;
    &lt;/core&gt;
    &lt;core xmlns=&quot;urn:activemq:core&quot;&gt;
      &lt;jmx-management-enabled&gt;true&lt;/jmx-management-enabled&gt;
    &lt;/core&gt;
  &lt;/configuration&gt;
  </code></pre>
  <p>With this setup, I still got the same result, after the the artemis node crash and comeback, the left over message was not moved to the other node.</p>
  <p>Update 2
  I tried to use non-wildcard queue as Justin suggested but still got the same behavior. One different I noticed is that if I use the non-wildcard queue, the consumer count is only 1 compare to 3 in the case of wildcard queue.Here is the attributes of the old queue after the crash</p>
  <pre><code>Acknowledge attempts    0
  Address QUEUE.service2.tech-drive2.188100000059.thai.notification
  Configuration managed   false
  Consumer count  0
  Consumers before dispatch   0
  Dead letter address DLQ
  Delay before dispatch   -1
  Delivering count    0
  Delivering size 0
  Durable true
  Durable delivering count    0
  Durable delivering size 0
  Durable message count   15
  Durable persistent size 102245
  Durable scheduled count 0
  Durable scheduled size  0
  Enabled true
  Exclusive   false
  Expiry address  ExpiryQueue
  Filter  
  First message age   840031
  First message as json   [{&quot;JMSType&quot;:&quot;service2&quot;,&quot;address&quot;:&quot;QUEUE.service2.tech-drive2.188100000059.thai.notification&quot;,&quot;messageID&quot;:8739,&quot;sentAt&quot;:1621969900922,&quot;accessToken&quot;:&quot;DONOTDISPLAY&quot;,&quot;type&quot;:3,&quot;priority&quot;:4,&quot;userID&quot;:&quot;ID:09502dc0-bd8d-11eb-b75c-c6609f1332c9&quot;,&quot;_AMQ_GROUP_ID&quot;:&quot;tech-drive2-188100000059-thai&quot;,&quot;sentBy&quot;:&quot;user@email.com&quot;,&quot;durable&quot;:true,&quot;JMSReplyTo&quot;:&quot;queue://QUEUE.service2.tech-drive2.188100000059.thai.notification&quot;,&quot;__AMQ_CID&quot;:&quot;c292b418-bd8b-11eb-b75c-c6609f1332c9&quot;,&quot;sentFrom&quot;:&quot;service2&quot;,&quot;originalDestination&quot;:&quot;QUEUE.service2.tech-drive2.188100000059.thai.notification&quot;,&quot;_AMQ_ROUTING_TYPE&quot;:1,&quot;JMSCorrelationID&quot;:&quot;90b783d0-d9cc-4188-9c9e-3453786b2105&quot;,&quot;expiration&quot;:0,&quot;timestamp&quot;:1621969900922}]
  First message timestamp 1621969900922
  Group buckets   -1
  Group count 0
  Group first key 
  Group rebalance false
  Group rebalance pause dispatch  false
  Id  606
  Last value  false
  Last value key  
  Max consumers   -1
  Message count   15
  Messages acknowledged   0
  Messages added  15
  Messages expired    0
  Messages killed 0
  Name    QUEUE.service2.tech-drive2.188100000059.thai.notification
  Object Name org.apache.activemq.artemis:broker=&quot;activemq-artemis-master-0&quot;,component=addresses,address=&quot;QUEUE.service2.tech-drive2.188100000059.thai.notification&quot;,subcomponent=queues,routing-type=&quot;anycast&quot;,queue=&quot;QUEUE.service2.tech-drive2.188100000059.thai.notification&quot;
  Paused  false
  Persistent size 102245
  Prepared transaction message count  0
  Purge on no consumers   false
  Retroactive resource    false
  Ring size   -1
  Routing type    ANYCAST
  Scheduled count 0
  Scheduled size  0
  Temporary   false
  User    6e25e08b-9587-40a3-b7e9-146360539258
  </code></pre>
  <p>and here is the attributes of the new queue</p>
  <pre><code>Attribute   Value
  Acknowledge attempts    0
  Address QUEUE.service2.tech-drive2.188100000059.thai.notification
  Configuration managed   false
  Consumer count  1
  Consumers before dispatch   0
  Dead letter address DLQ
  Delay before dispatch   -1
  Delivering count    0
  Delivering size 0
  Durable true
  Durable delivering count    0
  Durable delivering size 0
  Durable message count   0
  Durable persistent size 0
  Durable scheduled count 0
  Durable scheduled size  0
  Enabled true
  Exclusive   false
  Expiry address  ExpiryQueue
  Filter  
  First message age   
  First message as json   [{}]
  First message timestamp 
  Group buckets   -1
  Group count 0
  Group first key 
  Group rebalance false
  Group rebalance pause dispatch  false
  Id  866
  Last value  false
  Last value key  
  Max consumers   -1
  Message count   0
  Messages acknowledged   0
  Messages added  0
  Messages expired    0
  Messages killed 0
  Name    QUEUE.service2.tech-drive2.188100000059.thai.notification
  Object Name org.apache.activemq.artemis:broker=&quot;activemq-artemis-master-1&quot;,component=addresses,address=&quot;QUEUE.service2.tech-drive2.188100000059.thai.notification&quot;,subcomponent=queues,routing-type=&quot;anycast&quot;,queue=&quot;QUEUE.service2.tech-drive2.188100000059.thai.notification&quot;
  Paused  false
  Persistent size 0
  Prepared transaction message count  0
  Purge on no consumers   false
  Retroactive resource    false
  Ring size   -1
  Routing type    ANYCAST
  Scheduled count 0
  Scheduled size  0
  Temporary   false
  User    6e25e08b-9587-40a3-b7e9-146360539258
  </code></pre>