title: Configuring ActiveMQ Artemis in Wildfly 12 for time-critical delivery of messages in a cluster
body: <p>I can't figure out how to get the messaging subsystem in Wildfly 12 to redeliver a queue message to a different node when the original node fails. I need a way to direct a message to a different node when the first attempted node doesn't acknowledge/commit soon enough.</p>
  
  <p>I have a 3-node Wildfly 12 cluster with a single queue (TestQueue).  I deploy an application with a single bean that grabs a JMS connection and creates a session with a consumer on that queue; here's the constructor:</p>
  
  <pre><code>public TestMessageListener( ConnectionFactory connectionFactory, Destination destination )
  {
      this.context = connectionFactory.createContext( JMSContext.DUPS_OK_ACKNOWLEDGE );
      this.consumer = this.context.createConsumer( destination );
      this.consumer.setMessageListener( this );
      this.context.start();
  }
  </code></pre>
  
  <p>The connection factory and destination are injected elsewhere:</p>
  
  <pre><code>@Resource( lookup = "java:/ConnectionFactory" ) private ConnectionFactory connectionFactory;
  @Resource( lookup = "java:/jms/queue/TestQueue" ) private Destination destination;
  </code></pre>
  
  <p>Back in the listener I just log what it receives:</p>
  
  <pre><code>@Override
  public void onMessage( Message message )
  {
      try
      {
          message.acknowledge();
          String body = new String( message.getBody( byte[].class ), StandardCharsets.UTF_8 );
          LOG.info( body );
      }
      catch ( JMSException e )
      {
          LOG.warning( e.toString() );
      }
  }
  </code></pre>
  
  <p>Finally, I have STOMP enabled in the messaging subsystem configuration:</p>
  
  <pre><code>&lt;socket-binding-groups&gt;
      &lt;socket-binding-group name="full-ha-sockets" default-interface="public"&gt;
          &lt;socket-binding name="stomp" port="6164"/&gt;
          ... 
      &lt;/socket-binding-group&gt;
  &lt;/socket-binding-groups&gt;
  
  &lt;subsystem xmlns="urn:jboss:domain:messaging-activemq:3.0"&gt;
      &lt;server name="default"&gt;
          &lt;remote-acceptor name="stomp-acceptor" socket-binding="stomp"&gt;
              &lt;param name="protocols" value="STOMP"/&gt;
          &lt;/remote-acceptor&gt;            
          &lt;address-setting name="jms.queue.TestQueue" redistribution-delay="0"/&gt;
          ...
      &lt;/server&gt;
  </code></pre>
  
  <p>I connect over stomp and send a test message every 2 seconds with a unique identifier.  Each of the 3 nodes receives one in turn, round-robin.  Then I unplug the network cable from one of the nodes.</p>
  
  <p>After 1 minute (which I assume is connection-ttl), I get error messages on the other 2 nodes about connection failures:</p>
  
  <pre><code>2018-07-11 20:02:18,813 INFO  [TestMessageListener] (Thread-1 (ActiveMQ-client-global-threads)) TEST 435
  2018-07-11 20:02:21,448 WARN  [org.apache.activemq.artemis.core.client] (Thread-8 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$3@3070595f)) AMQ212037: Connection failure has been detected: AMQ119014: Did not receive data from /192.168.1.82:51046 within the 60,000ms connection TTL. The connection will now be closed. [code=CONNECTION_TIMEDOUT]
  2018-07-11 20:02:21,449 WARN  [org.apache.activemq.artemis.core.server] (Thread-8 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$3@3070595f)) AMQ222061: Client connection failed, clearing up resources for session b7be7d58-855c-11e8-91dd-6c626d5557a6
  2018-07-11 20:02:21,449 WARN  [org.apache.activemq.artemis.core.server] (Thread-8 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$3@3070595f)) AMQ222107: Cleared up resources for session b7be7d58-855c-11e8-91dd-6c626d5557a6
  2018-07-11 20:02:21,449 WARN  [org.apache.activemq.artemis.core.server] (Thread-8 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$3@3070595f))     AMQ222061: Client connection failed, clearing up resources for session b7becb79-855c-11e8-91dd-6c626d5557a6
  2018-07-11 20:02:21,449 WARN  [org.apache.activemq.artemis.core.server] (Thread-8 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$3@3070595f)) AMQ222107: Cleared up resources for session b7becb79-855c-11e8-91dd-6c626d5557a6
  </code></pre>
  
  <p>After an additional 30 seconds, I get another round of error messages about connection failures:</p>
  
  <pre><code>2018-07-11 20:02:49,443 WARN  [org.apache.activemq.artemis.core.client] (Thread-1 (ActiveMQ-client-global-threads)) AMQ212037: Connection failure has been detected: AMQ119011: Did not receive data from server for org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnection@5e696d4b[local= /192.168.1.27:39202, remote=/192.168.1.82:8080] [code=CONNECTION_TIMEDOUT]
  2018-07-11 20:02:49,444 WARN  [org.apache.activemq.artemis.core.server] (Thread-1 (ActiveMQ-client-global-threads)) AMQ222095: Connection failed with failedOver=false
  2018-07-11 20:02:49,446 WARN  [org.apache.activemq.artemis.core.server] (Thread-1 (ActiveMQ-client-global-threads)) AMQ222095: Connection failed with failedOver=false
  </code></pre>
  
  <p>Note that my STOMP client is connected to one of the good nodes and continues to send messages to the queue while the "failed" box is disconnected.</p>
  
  <p>My problems with this are:</p>
  
  <ul>
  <li>During the 90 seconds, Artemis continues to deliver messages to the unplugged box.</li>
  <li>I can't figure out how to get Artemis to try redelivering old messages to a different node even after the 90 seconds has elapsed.</li>
  <li>I don't understand why it continues to try delivering messages to the unplugged box after the first round of connection errors at 60s.</li>
  <li>Setting redistribution-delay has no effect, though I thought it would be useful due to <a href="https://activemq.apache.org/artemis/docs/latest/clusters.html" rel="nofollow noreferrer">https://activemq.apache.org/artemis/docs/latest/clusters.html</a>.</li>
  </ul>
  
  <p>Like this:</p>
  
  <pre><code>&lt;address-setting name="jms.queue.TestQueue" redistribution-delay="0"/&gt;
  </code></pre>
  
  <ul>
  <li>When I plug the network cable back in, all messages that should have been delivered to the failed node are now delivered.</li>
  </ul>
  
  <p>For this particular queue I would not only like redelivery attempts to another node, but I'd like a message acknowledgement timeout to trigger this redelivery or, failing that, a small connection-ttl of 750-1000ms. If I set connection-ttl to even 15000ms, all connections between all nodes (even when the whole cluster is healthy) throw errors after 15000ms.  According to the documentation at <a href="https://activemq.apache.org/artemis/docs/latest/configuration-index.html" rel="nofollow noreferrer">https://activemq.apache.org/artemis/docs/latest/configuration-index.html</a>, this parameter is, "TTL for the Bridge. This should be greater than the ping period."  It's unclear what parameter the "ping period" is and it's even more unclear how such a parameter would map to the Wildfly subsystem configuration.  I'm assuming connection-ttl is here, where I set it to 15000:</p>
  
  <pre><code>&lt;cluster-connection name="my-cluster" address="jms" connector-name="http-connector" connection-ttl="60000" retry-interval-multiplier="1.5" max-retry-interval="60000" discovery-group="dg-group1"/&gt;
  </code></pre>
  
  <p>I'm perfectly fine with receiving and dealing with duplicate messages; I thought a combination of JMSContext.DUPS_OK_ACKNOWLEDGE and redistribution-delay="0" would solve at least the redelivery portion of it.</p>
  
  <p>I tried JMSContext.TRANSACTED and used JMSContext.commit() and JMSContext.rollback(). Obviously rollback() doesn't apply when the failed node is cut off from the rest of the cluster, but it's the only way I can see to trigger redelivery.</p>
  
  <p>I'm at the stage right now where I'm just tweaking a seemingly endless number of configuration parameters with little to no effect.  Any help would be greatly appreciated.</p>