title: Artemis crashed and messages/queues lost
body: <p>Any ideas what happened here? Can messages and queues be restored? How to use ActiveMQ Artemis data? All queues/messages are lost. Only DLQ/ExpiryQueue.</p>
  
  <ol>
  <li>We had to do a configuration change. So we started updating the Slave.</li>
  <li>To failover to the Slave to fix the Master we did a <code>./artemis-service stop</code> @16:32 </li>
  <li>Some warnings on Master but not much else.</li>
  <li>On the Slave we can see some timeouts and connection failures. Not sure if Slave took over the Queues and worked ok or not.</li>
  <li>@16:46:27 we did a <code>./artemis-service start</code> on the Master</li>
  <li>Seems like Artemis started moving/removing bindings/journals</li>
  <li>Master seems to fail to start with Connections issues @16:46:33</li>
  <li>Slave tries to restart again and take control @16:46:37</li>
  <li>Slave cleans out journals/bindings</li>
  </ol>
  
  <p>And now all queues are gone. Only DLQ and ExpiryQueue exists on Slave. Master is down.</p>
  
  <p>Can we restore from <code>oldreplica</code>? Might be wiped already?
  We have not yet ried to restore from Replicas. Is there a manual or KnowledgeBase?</p>
  
  <p>Master:
  A few of these: </p>
  
  <pre><code>16:32:15,847 WARN  [org.apache.activemq.artemis.core.server] AMQ222061: Client connection failed, clearing up resources for session ID:xxx.yyy.zzz-30305-1552451287008-17:1:-1 
  16:32:15,847 WARN  [org.apache.activemq.artemis.core.server] AMQ222107: Cleared up resources for session ID:xxx.yyy.zzz-30305-1552451287008-17:1:-1 
  ...
  16:32:16,088 INFO  [org.apache.activemq.hawtio.plugin.PluginContextListener] Destroyed artemis-plugin plugin 
  16:32:16,093 INFO  [org.apache.activemq.hawtio.branding.PluginContextListener] Destroyed activemq-branding plugin 
  16:32:16,104 INFO  [org.apache.activemq.artemis.core.server] AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.4.0 [06926557-2906-11e8-a15f-005056926b6e] stopped, uptime 4 days 3 hours
  </code></pre>
  
  <p>Meanwhile on the Slave:</p>
  
  <pre><code>16:32:15,867 INFO  [org.apache.activemq.artemis.core.server] AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote 
  16:32:15,868 INFO  [org.apache.activemq.artemis.core.server] AMQ221067: Waiting 30 seconds for quorum vote results. 
  16:32:15,869 INFO  [org.apache.activemq.artemis.core.server] AMQ221068: Received all quorum votes. 
  16:32:15,869 INFO  [org.apache.activemq.artemis.core.server] AMQ221071: Failing over based on quorum vote results. 
  16:32:15,889 WARN  [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119015: The connection was disconnected because of server shutdown [code=DISCONNECTED] 
  16:32:15,944 WARN  [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119015: The connection was disconnected because of server shutdown [code=DISCONNECTED] 
  16:32:15,999 INFO  [org.apache.activemq.artemis.core.server] AMQ221037: ActiveMQServerImpl::serverUUID=06926557-2906-11e8-a15f-005056926b6e to become 'live' 
  16:32:16,450 WARN  [org.apache.activemq.artemis.core.client] AMQ212004: Failed to connect to server. 
  16:32:34,825 INFO  [org.apache.activemq.artemis.core.server] AMQ221003: Deploying queue DLQ on address DLQ 
  16:32:34,825 INFO  [org.apache.activemq.artemis.core.server] AMQ221003: Deploying queue ExpiryQueue on address ExpiryQueue 
  16:32:35,156 INFO  [org.apache.activemq.artemis.core.server] AMQ221007: Server is now live 
  
  16:33:51,161 WARN  [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119014: Did not receive data from /aaa.bbb.ccc.ddd:53781 within the 60,000ms connection TTL. The connection will now be closed. [code=CONNECTION_TIMEDOUT] 
  16:34:10,061 INFO  [io.hawt.web.LoginServlet] hawtio login is using 1800 sec. HttpSession timeout 
  16:34:11,891 ERROR [org.apache.activemq.artemis.core.server] AMQ224088: Timeout (10 seconds) while handshaking has occurred. 
  16:34:52,912 ERROR [org.apache.activemq.artemis.core.server] AMQ224088: Timeout (10 seconds) while handshaking has occurred. 
  16:35:01,183 WARN  [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119014: Did not receive data from /aaa.bbb.ccc.ddd:47171 within the 60,000ms connection TTL. The connection will now be closed. [code=CONNECTION_TIMEDOUT] 
  16:36:01,191 WARN  [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119014: Did not receive data from /aaa.bbb.ccc.ddd:54700 within the 60,000ms connection TTL. The connection will now be closed. [code=CONNECTION_TIMEDOUT] 
  16:36:12,705 ERROR [org.apache.activemq.artemis.core.server] AMQ224088: Timeout (10 seconds) while handshaking has occurred. 
  16:36:53,104 ERROR [org.apache.activemq.artemis.core.server] AMQ224088: Timeout (10 seconds) while handshaking has occurred. 
  16:37:11,203 WARN  [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119014: Did not receive data from /aaa.bbb.ccc.ddd:16751 within the 60,000ms connection TTL. The connection will now be closed. [code=CONNECTION_TIMEDOUT] 
  16:38:11,209 WARN  [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119014: Did not receive data from /aaa.bbb.ccc.ddd:20634 within the 60,000ms connection TTL. The connection will now be closed. [code=CONNECTION_TIMEDOUT] 
  16:38:12,182 ERROR [org.apache.activemq.artemis.core.server] AMQ224088: Timeout (10 seconds) while handshaking has occurred. 
  16:39:21,216 WARN  [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119014: Did not receive data from /aaa.bbb.ccc.ddd:61541 within the 60,000ms connection TTL. The connection will now be closed. [code=CONNECTION_TIMEDOUT] 
  16:40:21,225 WARN  [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119014: Did not receive data from /aaa.bbb.ccc.ddd:49708 within the 60,000ms connection TTL. The connection will now be closed. [code=CONNECTION_TIMEDOUT] 
  16:41:18,720 ERROR [org.apache.activemq.artemis.core.server] AMQ224088: Timeout (10 seconds) while handshaking has occurred. 
  16:45:51,271 WARN  [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119014: Did not receive data from /aaa.bbb.ccc.ddd:20748 within the 60,000ms connection TTL. The connection will now be closed. [code=CONNECTION_TIMEDOUT] 
  16:46:32,151 INFO  [org.apache.activemq.artemis.core.server] AMQ221025: Replication: sending AIOSequentialFile:/opt/brokers/ActiveMQServer2/./data/journal/activemq-data-167.amq (size=10,485,760) to replica. 
  16:46:33,100 INFO  [org.apache.activemq.artemis.core.server] AMQ221025: Replication: sending AIOSequentialFile:/opt/brokers/ActiveMQServer2/./data/journal/activemq-data-168.amq (size=10,485,760) to replica. 
  16:46:33,182 WARN  [org.apache.activemq.artemis.core.server] AMQ222061: Client connection failed, clearing up resources for session ID:xxx.yyy.zzz-30305-1552451287008-19:1:-1 
  16:46:33,188 WARN  [org.apache.activemq.artemis.core.server] AMQ222107: Cleared up resources for session ID:xxx.yyy.zzz-30305-1552451287008-19:1:-1 
  16:46:33,199 WARN  [org.apache.activemq.artemis.core.server] AMQ222061: Client connection failed, clearing up resources for session 380d4fc6-4f13-11e9-b0fb-00505692a0af 
  16:46:33,288 INFO  [io.hawt.web.AuthenticationFilter] Destroying hawtio authentication filter 
  16:46:33,290 INFO  [io.hawt.HawtioContextListener] Destroying hawtio services 
  16:46:33,391 INFO  [org.apache.activemq.hawtio.plugin.PluginContextListener] Destroyed artemis-plugin plugin 
  16:46:33,397 INFO  [org.apache.activemq.hawtio.branding.PluginContextListener] Destroyed activemq-branding plugin 
  16:46:33,408 INFO  [org.apache.activemq.artemis.core.server] AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.4.0 [06926557-2906-11e8-a15f-005056926b6e] stopped, uptime 4 days 3 hours 
  
  
  16:46:37,163 INFO  [org.apache.activemq.artemis.integration.bootstrap] AMQ101000: Starting ActiveMQ Artemis Server 
  16:46:37,266 INFO  [org.apache.activemq.artemis.core.server] AMQ221000: backup Message Broker is starting with configuration Broker Configuration (clustered=true,journalDirectory=./data/journal,bindingsDirectory=./data/bindings,largeMessagesDirectory=./data/large-messages,pagingDirectory=./data/paging) 
  16:46:37,288 INFO  [org.apache.activemq.artemis.core.server] AMQ221055: There were too many old replicated folders upon startup, removing /opt/brokers/ActiveMQServer2/./data/bindings/oldreplica.45 
  16:46:37,292 INFO  [org.apache.activemq.artemis.core.server] AMQ222162: Moving data directory /opt/brokers/ActiveMQServer2/./data/bindings to /opt/brokers/ActiveMQServer2/./data/bindings/oldreplica.47 
  16:46:37,295 INFO  [org.apache.activemq.artemis.core.server] AMQ221055: There were too many old replicated folders upon startup, removing /opt/brokers/ActiveMQServer2/./data/journal/oldreplica.49 
  16:46:37,597 INFO  [org.apache.activemq.artemis.core.server] AMQ222162: Moving data directory /opt/brokers/ActiveMQServer2/./data/journal to /opt/brokers/ActiveMQServer2/./data/journal/oldreplica.51 
  16:46:37,628 INFO  [org.apache.activemq.artemis.core.server] AMQ221055: There were too many old replicated folders upon startup, removing /opt/brokers/ActiveMQServer2/./data/paging/oldreplica.45 
  16:46:37,852 INFO  [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized activemq-branding plugin 
  16:46:38,056 INFO  [org.apache.activemq.artemis.core.server] AMQ222162: Moving data directory /opt/brokers/ActiveMQServer2/./data/paging to /opt/brokers/ActiveMQServer2/./data/paging/oldreplica.47 
  16:46:38,091 INFO  [org.apache.activemq.artemis.core.server] AMQ221012: Using AIO Journal 
  16:46:38,157 INFO  [org.apache.activemq.artemis.core.server] AMQ221057: Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx). being defined as 2,147,483,648 
  16:46:38,252 INFO  [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE 
  16:46:38,252 INFO  [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-amqp-protocol]. Adding protocol support for: AMQP 
  16:46:38,253 INFO  [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-hornetq-protocol]. Adding protocol support for: HORNETQ 
  16:46:38,253 INFO  [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-mqtt-protocol]. Adding protocol support for: MQTT 
  16:46:38,253 INFO  [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-openwire-protocol]. Adding protocol support for: OPENWIRE 
  16:46:38,254 INFO  [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-stomp-protocol]. Adding protocol support for: STOMP 
  16:46:38,328 INFO  [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized artemis-plugin plugin 
  16:46:42,502 INFO  [io.hawt.HawtioContextListener] Initialising hawtio services 
  16:46:42,572 INFO  [io.hawt.web.JolokiaConfiguredAgentServlet] Jolokia overridden property: [key=policyLocation, value=file:/opt/brokers/ActiveMQServer2//etc/jolokia-access.xml] 
  16:46:42,617 INFO  [io.hawt.web.RBACMBeanInvoker] Using MBean [hawtio:type=security,area=jmx,rank=0,name=HawtioDummyJMXSecurity] for role based access control 
  16:46:42,788 INFO  [io.hawt.system.ProxyWhitelist] Initial proxy whitelist: [localhost, 127.0.0.1, aaa.bbb.ccc.50, nodep02.domain.local] 
  16:46:43,202 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server started at http://nodep02:8161
  16:46:43,202 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia REST API available at http://nodep02:8161/console/jolokia
  </code></pre>
  
  <p>Back on the Master </p>
  
  <pre><code>16:46:27,913 INFO  [org.apache.activemq.artemis.integration.bootstrap] AMQ101000: Starting ActiveMQ Artemis Server 
  16:46:27,999 INFO  [org.apache.activemq.artemis.core.server] AMQ221000: live Message Broker is starting with configuration Broker Configuration (clustered=true,journalDirectory=./data/journal,bindingsDirectory=./data/bindings,largeMessagesDirectory=./data/large-messages,pagingDirectory=./data/paging) 
  16:46:28,440 INFO  [org.apache.activemq.artemis.core.server] AMQ221055: There were too many old replicated folders upon startup, removing /opt/brokers/ActiveMQServer1/./data/bindings/oldreplica.18 
  16:46:28,447 INFO  [org.apache.activemq.artemis.core.server] AMQ222162: Moving data directory /opt/brokers/ActiveMQServer1/./data/bindings to /opt/brokers/ActiveMQServer1/./data/bindings/oldreplica.20 
  16:46:28,450 INFO  [org.apache.activemq.artemis.core.server] AMQ221055: There were too many old replicated folders upon startup, removing /opt/brokers/ActiveMQServer1/./data/journal/oldreplica.18 
  16:46:28,478 INFO  [org.apache.activemq.artemis.core.server] AMQ222162: Moving data directory /opt/brokers/ActiveMQServer1/./data/journal to /opt/brokers/ActiveMQServer1/./data/journal/oldreplica.20 
  16:46:28,482 INFO  [org.apache.activemq.artemis.core.server] AMQ221055: There were too many old replicated folders upon startup, removing /opt/brokers/ActiveMQServer1/./data/paging/oldreplica.18 
  16:46:28,517 INFO  [org.apache.activemq.artemis.core.server] AMQ222162: Moving data directory /opt/brokers/ActiveMQServer1/./data/paging to /opt/brokers/ActiveMQServer1/./data/paging/oldreplica.20 
  16:46:28,580 INFO  [org.apache.activemq.artemis.core.server] AMQ221012: Using AIO Journal 
  16:46:33,209 WARN  [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119015: The connection was disconnected because of server shutdown [code=DISCONNECTED] 
  16:46:33,266 WARN  [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119015: The connection was disconnected because of server shutdown [code=DISCONNECTED] 
  16:46:33,273 WARN  [org.apache.activemq.artemis.journal] File not opened, file=null: java.lang.NullPointerException: File not opened, file=null 
          at org.apache.activemq.artemis.core.io.aio.AIOSequentialFile.checkOpened(AIOSequentialFile.java:328) [artemis-journal-2.4.0.jar:2.4.0] 
          at org.apache.activemq.artemis.core.io.aio.AIOSequentialFile.writeDirect(AIOSequentialFile.java:242) [artemis-journal-2.4.0.jar:2.4.0] 
          at org.apache.activemq.artemis.core.io.AbstractSequentialFile$LocalBufferObserver.flushBuffer(AbstractSequentialFile.java:306) [artemis-journal-2.4.0.jar:2.4.0] 
          at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flushBatch(TimedBuffer.java:310) [artemis-journal-2.4.0.jar:2.4.0] 
          at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flush(TimedBuffer.java:281) [artemis-journal-2.4.0.jar:2.4.0] 
          at org.apache.activemq.artemis.core.io.AbstractSequentialFileFactory.flush(AbstractSequentialFileFactory.java:195) [artemis-journal-2.4.0.jar:2.4.0] 
          at org.apache.activemq.artemis.core.journal.impl.JournalImpl.flush(JournalImpl.java:2194) [artemis-journal-2.4.0.jar:2.4.0] 
          at org.apache.activemq.artemis.core.journal.impl.JournalImpl.stop(JournalImpl.java:2356) [artemis-journal-2.4.0.jar:2.4.0] 
          at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:266) [artemis-server-2.4.0.jar:2.4.0] 
          at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:203) [artemis-server-2.4.0.jar:2.4.0] 
          at org.apache.activemq.artemis.core.replication.ReplicationEndpoint.stop(ReplicationEndpoint.java:327) [artemis-server-2.4.0.jar:2.4.0] 
          at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stopComponent(ActiveMQServerImpl.java:1256) [artemis-server-2.4.0.jar:2.4.0] 
          at org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run(SharedNothingBackupActivation.java:259) [artemis-server-2.4.0.jar:2.4.0] 
          at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:2951) [artemis-server-2.4.0.jar:2.4.0] 
  16:46:33,318 ERROR [org.apache.activemq.artemis.core.server] AMQ224000: Failure in initialisation: ActiveMQIllegalStateException[errorType=ILLEGAL_STATE message=AMQ119026: Backup Server was not yet in sync with live] 
          at org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run(SharedNothingBackupActivation.java:314) [artemis-server-2.4.0.jar:2.4.0] 
          at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:2951) [artemis-server-2.4.0.jar:2.4.0] 
  16:46:33,995 INFO  [io.hawt.web.keycloak.KeycloakServlet] Keycloak integration is disabled 
  16:46:40,486 WARN  [org.apache.activemq.artemis.core.server] AMQ222040: Server is stopped 
  16:46:50,033 WARN  [org.apache.activemq.artemis.core.server] AMQ222040: Server is stopped 
  16:47:00,034 WARN  [org.apache.activemq.artemis.core.server] AMQ222040: Server is stopped 
  </code></pre>
  
  <p>Broker.xml (Master)</p>
  
  <pre><code>        &lt;?xml version="1.0"?&gt;
      &lt;configuration xmlns="urn:activemq" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd"&gt;
          &lt;core xmlns="urn:activemq:core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:activemq:core "&gt;
              &lt;name&gt;ActiveMQ1&lt;/name&gt;
              &lt;persistence-enabled&gt;true&lt;/persistence-enabled&gt;
              &lt;journal-type&gt;ASYNCIO&lt;/journal-type&gt;
              &lt;paging-directory&gt;./data/paging&lt;/paging-directory&gt;
              &lt;bindings-directory&gt;./data/bindings&lt;/bindings-directory&gt;
              &lt;journal-directory&gt;./data/journal&lt;/journal-directory&gt;
              &lt;large-messages-directory&gt;./data/large-messages&lt;/large-messages-directory&gt;
              &lt;journal-datasync&gt;true&lt;/journal-datasync&gt;
              &lt;journal-min-files&gt;2&lt;/journal-min-files&gt;
              &lt;journal-pool-files&gt;-1&lt;/journal-pool-files&gt;
              &lt;journal-file-size&gt;10M&lt;/journal-file-size&gt;
              &lt;journal-buffer-size&gt;33554432&lt;/journal-buffer-size&gt;
              &lt;!-- Size in bytes --&gt;
              &lt;journal-buffer-timeout&gt;128000&lt;/journal-buffer-timeout&gt;
              &lt;journal-max-io&gt;4096&lt;/journal-max-io&gt;
              &lt;connectors&gt;
                  &lt;connector name="artemis"&gt;tcp://nodep01:61616&lt;/connector&gt;
                  &lt;connector name="ActiveMQ2-Connector"&gt;tcp://nodep02:61616&lt;/connector&gt;
              &lt;/connectors&gt;
              &lt;disk-scan-period&gt;5000&lt;/disk-scan-period&gt;
              &lt;max-disk-usage&gt;100&lt;/max-disk-usage&gt;
              &lt;critical-analyzer&gt;true&lt;/critical-analyzer&gt;
              &lt;critical-analyzer-timeout&gt;120000&lt;/critical-analyzer-timeout&gt;
              &lt;critical-analyzer-check-period&gt;60000&lt;/critical-analyzer-check-period&gt;
              &lt;critical-analyzer-policy&gt;HALT&lt;/critical-analyzer-policy&gt;
              &lt;acceptors&gt;
                  &lt;acceptor name="artemis"&gt;tcp://nodep01:61616?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=CORE,AMQP,STOMP,HORNETQ,MQTT,OPENWIRE;useEpoll=true;amqpCredits=1000;amqpLowCredits=300&lt;/acceptor&gt;
                  &lt;acceptor name="amqp"&gt;tcp://nodep01:5672?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=AMQP;useEpoll=true;amqpCredits=1000;amqpMinCredits=300&lt;/acceptor&gt;
                  &lt;acceptor name="stomp"&gt;tcp://nodep01:61613?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=STOMP;useEpoll=true&lt;/acceptor&gt;
                  &lt;acceptor name="hornetq"&gt;tcp://nodep01:5445?protocols=HORNETQ,STOMP;useEpoll=true&lt;/acceptor&gt;
                  &lt;acceptor name="artemis+ssl"&gt;tcp://nodep01:61443?sslEnabled=true;keyStorePath=/opt/brokers/ActiveMQServer1/activemq.keystore;keyStorePassword=CCCCC;tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=CORE,AMQP,STOMP,HORNETQ,MQTT,OPENWIRE;useEpoll=true;amqpCredits=1000;amqpLowCredits=300&lt;/acceptor&gt;
              &lt;/acceptors&gt;
              &lt;cluster-user&gt;XXXXXXXX&lt;/cluster-user&gt;
              &lt;cluster-password&gt;YYYYYYY&lt;/cluster-password&gt;
              &lt;cluster-connections&gt;
                  &lt;cluster-connection name="ActiveMQClusterPROD"&gt;
                      &lt;connector-ref&gt;artemis&lt;/connector-ref&gt;
                      &lt;min-large-message-size&gt;33554432&lt;/min-large-message-size&gt;
                      &lt;!-- Size should be eqvialent to journal-buffer-size --&gt;
                      &lt;message-load-balancing&gt;ON_DEMAND&lt;/message-load-balancing&gt;
                      &lt;max-hops&gt;1&lt;/max-hops&gt;
                      &lt;static-connectors&gt;
                          &lt;connector-ref&gt;ActiveMQ2-Connector&lt;/connector-ref&gt;
                      &lt;/static-connectors&gt;
                  &lt;/cluster-connection&gt;
              &lt;/cluster-connections&gt;
              &lt;ha-policy&gt;
                  &lt;replication&gt;
                      &lt;master&gt;
                          &lt;check-for-live-server&gt;true&lt;/check-for-live-server&gt;
                      &lt;/master&gt;
                  &lt;/replication&gt;
              &lt;/ha-policy&gt;
              &lt;security-settings&gt;
                  &lt;security-setting match="#"&gt;
                      &lt;permission type="createNonDurableQueue" roles="amq" /&gt;
                      &lt;permission type="deleteNonDurableQueue" roles="amq" /&gt;
                      &lt;permission type="createDurableQueue" roles="amq" /&gt;
                      &lt;permission type="deleteDurableQueue" roles="amq" /&gt;
                      &lt;permission type="createAddress" roles="amq" /&gt;
                      &lt;permission type="deleteAddress" roles="amq" /&gt;
                      &lt;permission type="consume" roles="amq" /&gt;
                      &lt;permission type="browse" roles="amq" /&gt;
                      &lt;permission type="send" roles="amq" /&gt;
                      &lt;!-- we need this otherwise ./artemis data imp wouldn't work--&gt;
                      &lt;permission type="manage" roles="amq" /&gt;
                  &lt;/security-setting&gt;
              &lt;/security-settings&gt;
              &lt;address-settings&gt;
                  &lt;!-- if you define auto-create on certain queues, management has to be auto-create --&gt;
                  &lt;address-setting match="activemq.management#"&gt;
                      &lt;dead-letter-address&gt;DLQ&lt;/dead-letter-address&gt;
                      &lt;expiry-address&gt;ExpiryQueue&lt;/expiry-address&gt;
                      &lt;redelivery-delay&gt;0&lt;/redelivery-delay&gt;
                      &lt;!-- with -1 only the global-max-size is in use for limiting --&gt;
                      &lt;max-size-bytes&gt;-1&lt;/max-size-bytes&gt;
                      &lt;message-counter-history-day-limit&gt;10&lt;/message-counter-history-day-limit&gt;
                      &lt;address-full-policy&gt;PAGE&lt;/address-full-policy&gt;
                      &lt;auto-create-queues&gt;true&lt;/auto-create-queues&gt;
                      &lt;auto-create-addresses&gt;true&lt;/auto-create-addresses&gt;
                      &lt;auto-create-jms-queues&gt;true&lt;/auto-create-jms-queues&gt;
                      &lt;auto-create-jms-topics&gt;true&lt;/auto-create-jms-topics&gt;
                  &lt;/address-setting&gt;
                  &lt;!--default for catch all--&gt;
                  &lt;address-setting match="#"&gt;
                      &lt;dead-letter-address&gt;DLQ&lt;/dead-letter-address&gt;
                      &lt;expiry-address&gt;ExpiryQueue&lt;/expiry-address&gt;
                      &lt;redelivery-delay&gt;0&lt;/redelivery-delay&gt;
                      &lt;!-- with -1 only the global-max-size is in use for limiting --&gt;
                      &lt;max-size-bytes&gt;-1&lt;/max-size-bytes&gt;
                      &lt;message-counter-history-day-limit&gt;10&lt;/message-counter-history-day-limit&gt;
                      &lt;address-full-policy&gt;PAGE&lt;/address-full-policy&gt;
                      &lt;auto-create-queues&gt;true&lt;/auto-create-queues&gt;
                      &lt;auto-create-addresses&gt;true&lt;/auto-create-addresses&gt;
                      &lt;auto-create-jms-queues&gt;true&lt;/auto-create-jms-queues&gt;
                      &lt;auto-create-jms-topics&gt;true&lt;/auto-create-jms-topics&gt;
                  &lt;/address-setting&gt;
              &lt;/address-settings&gt;
              &lt;addresses&gt;
                  &lt;address name="DLQ"&gt;
                      &lt;anycast&gt;
                          &lt;queue name="DLQ" /&gt;
                      &lt;/anycast&gt;
                  &lt;/address&gt;
                  &lt;address name="ExpiryQueue"&gt;
                      &lt;anycast&gt;
                          &lt;queue name="ExpiryQueue" /&gt;
                      &lt;/anycast&gt;
                  &lt;/address&gt;
              &lt;/addresses&gt;
          &lt;/core&gt;
      &lt;/configuration&gt;
  </code></pre>
  
  <p>Broker.xml (Slave)</p>
  
  <pre><code>    &lt;?xml version="1.0"?&gt;
      &lt;configuration xmlns="urn:activemq" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd"&gt;
          &lt;core xmlns="urn:activemq:core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:activemq:core "&gt;
              &lt;name&gt;ActiveMQ2&lt;/name&gt;
              &lt;persistence-enabled&gt;true&lt;/persistence-enabled&gt;
              &lt;journal-type&gt;ASYNCIO&lt;/journal-type&gt;
              &lt;paging-directory&gt;./data/paging&lt;/paging-directory&gt;
              &lt;bindings-directory&gt;./data/bindings&lt;/bindings-directory&gt;
              &lt;journal-directory&gt;./data/journal&lt;/journal-directory&gt;
              &lt;large-messages-directory&gt;./data/large-messages&lt;/large-messages-directory&gt;
              &lt;journal-datasync&gt;true&lt;/journal-datasync&gt;
              &lt;journal-min-files&gt;2&lt;/journal-min-files&gt;
              &lt;journal-pool-files&gt;-1&lt;/journal-pool-files&gt;
              &lt;journal-file-size&gt;10M&lt;/journal-file-size&gt;
              &lt;journal-buffer-size&gt;33554432&lt;/journal-buffer-size&gt;
              &lt;!-- size in bytes --&gt;
              &lt;journal-buffer-timeout&gt;104000&lt;/journal-buffer-timeout&gt;
              &lt;journal-max-io&gt;4096&lt;/journal-max-io&gt;
              &lt;connectors&gt;
                  &lt;!-- Connector used to be announced through cluster connections and notifications --&gt;
                  &lt;connector name="artemis"&gt;tcp://nodep02:61616&lt;/connector&gt;
                  &lt;connector name="ActiveMQ1-Connector"&gt;tcp://nodep01:61616&lt;/connector&gt;
              &lt;/connectors&gt;
              &lt;disk-scan-period&gt;5000&lt;/disk-scan-period&gt;
              &lt;max-disk-usage&gt;100&lt;/max-disk-usage&gt;
              &lt;critical-analyzer&gt;true&lt;/critical-analyzer&gt;
              &lt;critical-analyzer-timeout&gt;120000&lt;/critical-analyzer-timeout&gt;
              &lt;critical-analyzer-check-period&gt;60000&lt;/critical-analyzer-check-period&gt;
              &lt;critical-analyzer-policy&gt;HALT&lt;/critical-analyzer-policy&gt;
              &lt;acceptors&gt;
                  &lt;acceptor name="artemis"&gt;tcp://nodep02:61616?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=CORE,AMQP,STOMP,HORNETQ,MQTT,OPENWIRE;useEpoll=true;amqpCredits=1000;amqpLowCredits=300&lt;/acceptor&gt;
                  &lt;acceptor name="amqp"&gt;tcp://nodep02:5672?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=AMQP;useEpoll=true;amqpCredits=1000;amqpMinCredits=300&lt;/acceptor&gt;
                  &lt;acceptor name="stomp"&gt;tcp://nodep02:61613?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=STOMP;useEpoll=true&lt;/acceptor&gt;
                  &lt;acceptor name="hornetq"&gt;tcp://nodep02:5445?protocols=HORNETQ,STOMP;useEpoll=true&lt;/acceptor&gt;
                  &lt;acceptor name="mqtt"&gt;tcp://nodep02:1883?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=MQTT;useEpoll=true&lt;/acceptor&gt;
                  &lt;acceptor name="artemis+ssl"&gt;tcp://nodep02:61443?sslEnabled=true;keyStorePath=/opt/brokers/ActiveMQServer2/activemq.keystore;keyStorePassword=CCCCC;tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=CORE,AMQP,STOMP,HORNETQ,MQTT,OPENWIRE;useEpoll=true;amqpCredits=1000;amqpLowCredits=300&lt;/acceptor&gt;
              &lt;/acceptors&gt;
              &lt;cluster-user&gt;XXXXXXXX&lt;/cluster-user&gt;
              &lt;cluster-password&gt;YYYYYYY&lt;/cluster-password&gt;
              &lt;cluster-connections&gt;
                  &lt;cluster-connection name="ActiveMQClusterPROD"&gt;
                      &lt;connector-ref&gt;artemis&lt;/connector-ref&gt;
                      &lt;min-large-message-size&gt;33554432&lt;/min-large-message-size&gt;
                      &lt;message-load-balancing&gt;ON_DEMAND&lt;/message-load-balancing&gt;
                      &lt;max-hops&gt;1&lt;/max-hops&gt;
                      &lt;static-connectors&gt;
                          &lt;connector-ref&gt;ActiveMQ1-Connector&lt;/connector-ref&gt;
                      &lt;/static-connectors&gt;
                  &lt;/cluster-connection&gt;
              &lt;/cluster-connections&gt;
              &lt;ha-policy&gt;
                  &lt;replication&gt;
                      &lt;slave&gt;
                          &lt;allow-failback&gt;true&lt;/allow-failback&gt;
                      &lt;/slave&gt;
                  &lt;/replication&gt;
              &lt;/ha-policy&gt;
              &lt;security-settings&gt;
                  &lt;security-setting match="#"&gt;
                      &lt;permission type="createNonDurableQueue" roles="amq" /&gt;
                      &lt;permission type="deleteNonDurableQueue" roles="amq" /&gt;
                      &lt;permission type="createDurableQueue" roles="amq" /&gt;
                      &lt;permission type="deleteDurableQueue" roles="amq" /&gt;
                      &lt;permission type="createAddress" roles="amq" /&gt;
                      &lt;permission type="deleteAddress" roles="amq" /&gt;
                      &lt;permission type="consume" roles="amq" /&gt;
                      &lt;permission type="browse" roles="amq" /&gt;
                      &lt;permission type="send" roles="amq" /&gt;
                      &lt;!-- we need this otherwise ./artemis data imp wouldn't work--&gt;
                      &lt;permission type="manage" roles="amq" /&gt;
                  &lt;/security-setting&gt;
              &lt;/security-settings&gt;
              &lt;address-settings&gt;
                  &lt;!-- if you define auto-create on certain queues, management has to be auto-create --&gt;
                  &lt;address-setting match="activemq.management#"&gt;
                      &lt;dead-letter-address&gt;DLQ&lt;/dead-letter-address&gt;
                      &lt;expiry-address&gt;ExpiryQueue&lt;/expiry-address&gt;
                      &lt;redelivery-delay&gt;0&lt;/redelivery-delay&gt;
                      &lt;!-- with -1 only the global-max-size is in use for limiting --&gt;
                      &lt;max-size-bytes&gt;-1&lt;/max-size-bytes&gt;
                      &lt;message-counter-history-day-limit&gt;10&lt;/message-counter-history-day-limit&gt;
                      &lt;address-full-policy&gt;PAGE&lt;/address-full-policy&gt;
                      &lt;auto-create-queues&gt;true&lt;/auto-create-queues&gt;
                      &lt;auto-create-addresses&gt;true&lt;/auto-create-addresses&gt;
                      &lt;auto-create-jms-queues&gt;true&lt;/auto-create-jms-queues&gt;
                      &lt;auto-create-jms-topics&gt;true&lt;/auto-create-jms-topics&gt;
                  &lt;/address-setting&gt;
                  &lt;!--default for catch all--&gt;
                  &lt;address-setting match="#"&gt;
                      &lt;dead-letter-address&gt;DLQ&lt;/dead-letter-address&gt;
                      &lt;expiry-address&gt;ExpiryQueue&lt;/expiry-address&gt;
                      &lt;redelivery-delay&gt;0&lt;/redelivery-delay&gt;
                      &lt;!-- with -1 only the global-max-size is in use for limiting --&gt;
                      &lt;max-size-bytes&gt;-1&lt;/max-size-bytes&gt;
                      &lt;message-counter-history-day-limit&gt;10&lt;/message-counter-history-day-limit&gt;
                      &lt;address-full-policy&gt;PAGE&lt;/address-full-policy&gt;
                      &lt;auto-create-queues&gt;true&lt;/auto-create-queues&gt;
                      &lt;auto-create-addresses&gt;true&lt;/auto-create-addresses&gt;
                      &lt;auto-create-jms-queues&gt;true&lt;/auto-create-jms-queues&gt;
                      &lt;auto-create-jms-topics&gt;true&lt;/auto-create-jms-topics&gt;
                  &lt;/address-setting&gt;
              &lt;/address-settings&gt;
              &lt;addresses&gt;
                  &lt;address name="DLQ"&gt;
                      &lt;anycast&gt;
                          &lt;queue name="DLQ" /&gt;
                      &lt;/anycast&gt;
                  &lt;/address&gt;
                  &lt;address name="ExpiryQueue"&gt;
                      &lt;anycast&gt;
                          &lt;queue name="ExpiryQueue" /&gt;
                      &lt;/anycast&gt;
                  &lt;/address&gt;
              &lt;/addresses&gt;
          &lt;/core&gt;
      &lt;/configuration&gt;
  </code></pre>