title: How to avoid blocking of queue browsing after ActiveMQ checkpoint call
body: <p>There's a problem when Using ActiveMQ with a large number of Persistence Queues (250) á 1000 persistent TextMessages á 10 KB.</p>
  
  <p>A scenario requires these messages to remain in the storage over a long time (days), until they are consumed (large amounts of data are staged for distribution for many consumer, that could be offline for some days).</p>
  
  <p>After the Persistence Store is filled with these Messages and after a broker restart we can browse/consume some Queues  <em>until</em> the #checkpoint call after 30 seconds.</p>
  
  <p>This call causes the broker to use all available memory and never releases it for other tasks such as Queue browse/consume. Internally the MessageCursor seems to decide, that there is not enough memory and stops delivery of queue content to browsers/consumers.</p>
  
  <p><strong>=> Is there a way to avoid this behaviour by configuration or is this a bug?</strong></p>
  
  <p>The expectation is, that we can consume/browse any queue under all circumstances.</p>
  
  <p>Settings below are in production for some time now and several recommendations are applied found in the ActiveMQ documentation (destination policies, systemUsage, persistence store options etc.)</p>
  
  <ul>
  <li>Behaviour is tested with ActiveMQ: 5.11.2, 5.13.0 and 5.5.1.</li>
  <li>Memory Settings: Xmx=1024m</li>
  <li>Java: 1.8 or 1.7</li>
  <li>OS: Windows, MacOS, Linux</li>
  <li>PersistenceAdapter: KahaDB or LevelDB</li>
  <li>Disc: enough free space (200 GB) and physical memory (16 GB max).</li>
  </ul>
  
  <p>Besides the above mentioned settings we use the following settings for the broker (btw: changing the memoryLimit to a lower value like 1mb does not change the situation):</p>
  
  <pre><code>&lt;destinationPolicy&gt;
      &lt;policyMap&gt;
          &lt;policyEntries&gt;
              &lt;policyEntry queue="&gt;" producerFlowControl="false" optimizedDispatch="true" memoryLimit="128mb" timeBeforeDispatchStarts="1000"&gt;
                  &lt;dispatchPolicy&gt;
                      &lt;strictOrderDispatchPolicy /&gt;
                  &lt;/dispatchPolicy&gt;
                  &lt;pendingQueuePolicy&gt;
                      &lt;storeCursor /&gt;
                  &lt;/pendingQueuePolicy&gt;
              &lt;/policyEntry&gt;
          &lt;/policyEntries&gt;
      &lt;/policyMap&gt;
  &lt;/destinationPolicy&gt;
  &lt;systemUsage&gt;
      &lt;systemUsage sendFailIfNoSpace="true"&gt;
          &lt;memoryUsage&gt;
              &lt;memoryUsage limit="500 mb" /&gt;
          &lt;/memoryUsage&gt;
          &lt;storeUsage&gt;
              &lt;storeUsage limit="80000 mb" /&gt;
          &lt;/storeUsage&gt;
          &lt;tempUsage&gt;
              &lt;tempUsage limit="1000 mb" /&gt;
          &lt;/tempUsage&gt;
      &lt;/systemUsage&gt;
  &lt;/systemUsage&gt;
  </code></pre>
  
  <p>If we set the <strong>cursorMemoryHighWaterMark</strong> in the destinationPolicy to a higher value like <strong>150</strong> or <strong>600</strong> depending on the difference between memoryUsage and the available heap space relieves the situation a bit for a workaround, but this is not really an option for production systems in my point of view.</p>
  
  <p>Screenie with information from Oracle Mission Control showing those ActiveMQTextMessage instances that are never released from memory: </p>
  
  <p><a href="http://goo.gl/EjEixV" rel="nofollow">http://goo.gl/EjEixV</a></p>