title: Transaction not prepared (ActiveMQ), "the XA resource has become unavailable" (Atomikos), endless exceptions in log
body: <p>we have a problem with Atomikos (TransactionManager) and ActiveMQ. Atomikos is used in a Spring environment to enable XA transactions. We tested the failover behavior of the ActiveMQ client side and notice some exceptions that are not going away. Our test scenario is the following:</p>
  
  <ul>
  <li>start two brokers that use the same database, one broker will be the master, the second the slave</li>
  <li>start the client; client will connect to the master broker</li>
  <li>stop the master broker abruptly (kill -9)</li>
  <li>failover to the second broker happens and this works</li>
  </ul>
  
  <p>We use the following versions:</p>
  
  <ul>
  <li>ActiveMQ 5.10.0</li>
  <li>Atomikos 3.9.3, 4.0.0M3, 3.9.7</li>
  <li>Spring 3.2.4.RELEASE</li>
  </ul>
  
  <p>Now we get the following exception from Atomikos:</p>
  
  <pre><code>2014-06-12 11:28:42 tpe-1 com.atomikos.datasource.xa.XAResourceTransaction WARN  - XA resource 'connectionFactoryOut': commit for XID '6170695F3134303235363533313236323230303334383030303031:6170695F31343032353635333132363232333438' raised -7: the XA resource has become unavailable
  javax.transaction.xa.XAException: The JMS connection has failed: java.io.EOFException
      at org.apache.activemq.TransactionContext.toXAException(TransactionContext.java:793)
      at org.apache.activemq.TransactionContext.commit(TransactionContext.java:590)
      at com.atomikos.datasource.xa.XAResourceTransaction.commit(XAResourceTransaction.java:733)
      at com.atomikos.icatch.imp.CommitMessage.send(CommitMessage.java:72)
      at com.atomikos.icatch.imp.PropagationMessage.submit(PropagationMessage.java:83)
      at com.atomikos.icatch.imp.Propagator$PropagatorThread.run(Propagator.java:79)
      at com.atomikos.icatch.imp.Propagator.submitPropagationMessage(Propagator.java:58)
      at com.atomikos.icatch.imp.CoordinatorStateHandler.commitFromWithinCallback(CoordinatorStateHandler.java:582)
      at com.atomikos.icatch.imp.ActiveStateHandler$6.doCommit(ActiveStateHandler.java:301)
      at com.atomikos.icatch.imp.CoordinatorStateHandler.commitWithAfterCompletionNotification(CoordinatorStateHandler.java:852)
      at com.atomikos.icatch.imp.ActiveStateHandler.commit(ActiveStateHandler.java:296)
      at com.atomikos.icatch.imp.CoordinatorImp.commit(CoordinatorImp.java:707)
      at com.atomikos.icatch.imp.CoordinatorImp.terminate(CoordinatorImp.java:968)
      at com.atomikos.icatch.imp.CompositeTerminatorImp.commit(CompositeTerminatorImp.java:82)
      at com.atomikos.icatch.imp.CompositeTransactionImp.commit(CompositeTransactionImp.java:336)
      at com.atomikos.icatch.jta.TransactionImp.commit(TransactionImp.java:190)
      at com.atomikos.icatch.jta.TransactionManagerImp.commit(TransactionManagerImp.java:436)
      at com.atomikos.icatch.jta.UserTransactionImp.commit(UserTransactionImp.java:107)
      at org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1021)
      at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:757)
      at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:726)
      at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:150)
      at net.sprd.messaging.test.data_generator.SenderThread$1.run(SenderThread.java:42)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)
  Caused by: org.apache.activemq.ConnectionFailedException: The JMS connection has failed: java.io.EOFException
      at org.apache.activemq.ActiveMQConnection.checkClosedOrFailed(ActiveMQConnection.java:1483)
      at org.apache.activemq.TransactionContext.commit(TransactionContext.java:551)
      ... 26 more
  Caused by: java.io.EOFException
      at java.io.DataInputStream.readInt(DataInputStream.java:392)
      at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:275)
      at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)
      at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)
      at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
      ... 1 more
  2014-06-12 11:28:42 tpe-1 com.atomikos.icatch.imp.CommitMessage WARN  - Unexpected error in commit
  com.atomikos.icatch.SysException: XA resource 'connectionFactoryOut': commit for XID '6170695F3134303235363533313236323230303334383030303031:6170695F31343032353635333132363232333438' raised -7: the XA resource has become unavailable
      at com.atomikos.datasource.xa.XAResourceTransaction.commit(XAResourceTransaction.java:773)
      at com.atomikos.icatch.imp.CommitMessage.send(CommitMessage.java:72)
      at com.atomikos.icatch.imp.PropagationMessage.submit(PropagationMessage.java:83)
      at com.atomikos.icatch.imp.Propagator$PropagatorThread.run(Propagator.java:79)
      at com.atomikos.icatch.imp.Propagator.submitPropagationMessage(Propagator.java:58)
      at com.atomikos.icatch.imp.CoordinatorStateHandler.commitFromWithinCallback(CoordinatorStateHandler.java:582)
      at com.atomikos.icatch.imp.ActiveStateHandler$6.doCommit(ActiveStateHandler.java:301)
      at com.atomikos.icatch.imp.CoordinatorStateHandler.commitWithAfterCompletionNotification(CoordinatorStateHandler.java:852)
      at com.atomikos.icatch.imp.ActiveStateHandler.commit(ActiveStateHandler.java:296)
      at com.atomikos.icatch.imp.CoordinatorImp.commit(CoordinatorImp.java:707)
      at com.atomikos.icatch.imp.CoordinatorImp.terminate(CoordinatorImp.java:968)
      at com.atomikos.icatch.imp.CompositeTerminatorImp.commit(CompositeTerminatorImp.java:82)
      at com.atomikos.icatch.imp.CompositeTransactionImp.commit(CompositeTransactionImp.java:336)
      at com.atomikos.icatch.jta.TransactionImp.commit(TransactionImp.java:190)
      at com.atomikos.icatch.jta.TransactionManagerImp.commit(TransactionManagerImp.java:436)
      at com.atomikos.icatch.jta.UserTransactionImp.commit(UserTransactionImp.java:107)
      at org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1021)
      at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:757)
      at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:726)
      at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:150)
      at net.sprd.messaging.test.data_generator.SenderThread$1.run(SenderThread.java:42)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)
  </code></pre>
  
  <p>The Spring configuration for the sending parts looks like this:
  
              
          </p>
  
  <pre><code>    &lt;bean id="jmsTemplateBroker" class="org.springframework.jms.core.JmsTemplate"&gt;
          &lt;constructor-arg name="connectionFactory" ref="cachingSendingSessionFactoryBroker" /&gt;
          &lt;property name="sessionTransacted" value="true" /&gt;
      &lt;/bean&gt;
  
      &lt;bean id="cachingSendingSessionFactoryBroker" class="org.springframework.jms.connection.CachingConnectionFactory"&gt;
          &lt;constructor-arg name="targetConnectionFactory" ref="atomikosConnectionFactoryBroker" /&gt;
          &lt;property name="sessionCacheSize" value="1000" /&gt;
          &lt;property name="cacheConsumers" value="false" /&gt;
          &lt;property name="cacheProducers" value="false" /&gt;
      &lt;/bean&gt;
  
      &lt;bean id="atomikosConnectionFactoryBroker" class="com.atomikos.jms.AtomikosConnectionFactoryBean"
          init-method="init" destroy-method="close" depends-on="activeMQConnectionFactoryBroker"&gt;
          &lt;property name="uniqueResourceName" value="atomikosConnectionFactoryBroker" /&gt;
          &lt;property name="xaConnectionFactory" ref="activeMQConnectionFactoryBroker" /&gt;
          &lt;property name="minPoolSize" value="1" /&gt;
          &lt;property name="maxPoolSize" value="1000" /&gt;
      &lt;/bean&gt;
  
      &lt;bean id="activeMQConnectionFactoryBroker" class="org.apache.activemq.ActiveMQXAConnectionFactory"&gt;
          &lt;property name="brokerURL" value="${broker.brokerURL}" /&gt;
          &lt;property name="userName" value="testUser" /&gt;
          &lt;property name="password" value="testPassword" /&gt;
          &lt;property name="redeliveryPolicy" ref="redeliveryPolicyBroker" /&gt;
          &lt;property name="prefetchPolicy" ref="activeMQPrefechPolicyBroker" /&gt;
      &lt;/bean&gt;
  
      &lt;bean id="activeMQPrefechPolicyBroker" class="org.apache.activemq.ActiveMQPrefetchPolicy"&gt;
          &lt;property name="all" value="100" /&gt;
      &lt;/bean&gt;
  
      &lt;bean id="redeliveryPolicyBroker" class="org.apache.activemq.RedeliveryPolicy"&gt;
          &lt;property name="initialRedeliveryDelay" value="1000" /&gt;
          &lt;property name="redeliveryDelay" value="1000" /&gt;
          &lt;property name="useExponentialBackOff" value="false" /&gt;
          &lt;property name="maximumRedeliveries" value="1000" /&gt;
      &lt;/bean&gt;
  </code></pre>
  
  <p>The Atomikos beans are configured as follows:
  
              
              
              
          </p>
  
  <pre><code>    &lt;bean id="atomikosTransactionManager" class="com.atomikos.icatch.jta.UserTransactionManager"
          init-method="init" destroy-method="close" depends-on="userTransactionService"&gt;
          &lt;property name="forceShutdown" value="true" /&gt;
      &lt;/bean&gt;
  
      &lt;bean id="atomikosUserTransaction" class="com.atomikos.icatch.jta.UserTransactionImp"&gt;
          &lt;property name="transactionTimeout" value="70" /&gt;
      &lt;/bean&gt;
  
  
      &lt;bean id="timestamp" class="java.lang.String"&gt;
          &lt;constructor-arg value="#{'' + T(java.lang.System).currentTimeMillis()}" /&gt;
      &lt;/bean&gt;
  
      &lt;bean id="userTransactionService" class="com.atomikos.icatch.config.UserTransactionServiceImp"
          init-method="init" destroy-method="shutdownForce"&gt;
          &lt;constructor-arg&gt;
              &lt;props&gt;
                  &lt;prop key="com.atomikos.icatch.service"&gt;com.atomikos.icatch.standalone.UserTransactionServiceFactory&lt;/prop&gt;
                  &lt;prop key="com.atomikos.icatch.output_dir"&gt;/tmp/atomikosOutput&lt;/prop&gt;
                  &lt;prop key="com.atomikos.icatch.console_file_name"&gt;/tmp/api_#{timestamp}.out&lt;/prop&gt;
                  &lt;prop key="com.atomikos.icatch.console_log_level"&gt;DEBUG&lt;/prop&gt;
                  &lt;prop key="com.atomikos.icatch.tm_unique_name"&gt;api_#{timestamp}&lt;/prop&gt;
                  &lt;prop key="com.atomikos.icatch.threaded_2pc"&gt;false&lt;/prop&gt;
                  &lt;prop key="com.atomikos.icatch.max_actives"&gt;-1&lt;/prop&gt;
                  &lt;prop key="com.atomikos.icatch.max_timeout"&gt;10000&lt;/prop&gt;
                  &lt;prop key="com.atomikos.icatch.default_jta_timeout"&gt;999&lt;/prop&gt;
                  &lt;prop key="com.atomikos.icatch.serial_jta_transactions"&gt;false&lt;/prop&gt;
                  &lt;prop key="com.atomikos.icatch.enable_logging"&gt;false&lt;/prop&gt;
                  &lt;prop key="com.atomikos.icatch.log_base_name"&gt;api#{timestamp}&lt;/prop&gt;
                  &lt;prop key="com.atomikos.icatch.log_base_dir"&gt;/tmp/atomikos&lt;/prop&gt;
                  &lt;prop key="com.atomikos.icatch.checkpoint_interval"&gt;5000&lt;/prop&gt;
              &lt;/props&gt;
          &lt;/constructor-arg&gt;
      &lt;/bean&gt;
  </code></pre>
  
  <p>These exceptions are thrown periodically. We assume the following: Atomikos sends a prepare statement to the ActiveMQ broker. After that the broker is shut down. Now the prepared statement is lost at the ActiveMQ side and so Atomikos will try to recover this gone transaction. Because both ActiveMQ broker use the same database we think that the prepared statement should be stored in the database. So the slave broker can resume the transaction. But this doesn't happen.</p>
  
  <p>Has anyone an idea if this is a bug of ActiveMQ or if there is a "hidden" configuration property of ActiveMQ to enable the prepared mode of XA transactions?</p>
  
  <p>Side mark: we make sure that we use unique ids for the transaction manager name (related to: <a href="https://stackoverflow.com/questions/6224353/activemq-jms-xa-atomikos-transaction-not-started-error">Stackoverflow article about Atomikos unique ids</a>)</p>
  
  <p>Thanks.</p>