title: Periods of prolonged inactivity and frequent MessageLockLostException in QueueClient
body: <h2>Background</h2>
  <p>We have a data transfer solution with Azure Service Bus as the message broker. We are transferring data from <code>x</code> datasets through <code>x</code> queues - with <code>x</code> dedicated <code>QueueClient</code>s as senders. Some senders publish messages at the rate of one message every two seconds, while others publish one every 15 minutes.</p>
  <p>The application on the data source side (where senders are) is working just fine, giving us the desired throughput.</p>
  <p>On the other side, we have an application with one QueueClient receiver per queue with the following configuration:</p>
  <ul>
  <li><code>maxConcurrentCalls</code> = <code>1</code></li>
  <li><code>autoComplete</code> = <code>true</code> (if receive mode = <code>RECEIVEANDDELETE</code>) and <code>false</code> (if receive mode = <code>PEEKLOCK</code>) - we have some receivers where, if they shut-down unexpectedly, would want to preserve the messages in the Service Bus Queue.</li>
  <li><code>maxAutoRenewDuration</code> = <code>3</code> minutes (lock duraition on all queues = <code>30</code> seconds)</li>
  <li>an Executor service with a single thread</li>
  </ul>
  <p>The <code>MessageHandler</code> registered with each of these receivers does the following:</p>
  <pre class="lang-java prettyprint-override"><code>public CompletableFuture&lt;Void&gt; onMessageAsync(final IMessage message) {
  
      // deserialize the message body
      final CustomObject customObject = (CustomObject)SerializationUtils.deserialize((byte[])message.getMessageBody().getBinaryData().get(0));
  
      // process processDB1() and processDB2() asynchronously
      final List&lt;CompletableFuture&lt;Boolean&gt;&gt; processFutures = new ArrayList&lt;CompletableFuture&lt;Boolean&gt;&gt;();
  
      processFutures.add(processDB1(customObject));  // processDB1() returns Boolean
      processFutures.add(processDB2(customObject)); // processDB2() returns Boolean
  
      // join both the completablefutures to get the result Booleans
      List&lt;Boolean&gt; results = CompletableFuture.allOf(processFutures.toArray(new CompletableFuture[processFutures.size()])).thenApply(future -&gt; processFutures.stream()
          .map(CompletableFuture&lt;Boolean&gt;::join).collect(Collectors.toList())
  
      if (results.contains(false)) {
          // dead-letter the message if results contains false
          return getQueueClient().deadLetterAsync(message.getLockToken());
      } else {
          // complete the message otherwise
          getQueueClient().completeAsync(message.getLockToken());
      }
  }
  
  </code></pre>
  <p>We tested with the following scenarios:</p>
  <h2>Scenario 1 - receive mode = <code>RECEIVEANDDELETE</code>, message publish rate: <code>30/ minute</code></h2>
  <h3>Expected Behavior</h3>
  <p>The messages should be received continuosuly with a constant throughput (which need not necessarily be the throughput at source, where messages are published).</p>
  <h3>Actual behavior</h3>
  <p>We observe random, long periods of inactivity from the QueueClient - ranging from minutes to hours - there is <strong>no Outgoing Messages</strong> from the Service Bus namespace (observed on the Metrics charts) and there are no consumption logs for the same time periods!</p>
  <h2>Scenario 2 - receive mode = <code>PEEKLOCK</code>, message publish rate: <code>30/ minute</code></h2>
  <h3>Expected Behavior</h3>
  <p>The messages should be received continuosuly with a constant throughput (which need not necessarily be the throughput at source, where messages are published).</p>
  <h3>Actual behavior</h3>
  <p>We keep seeing <code>MessageLockLostException</code> constantly after 20-30 minutes into the run of the application.</p>
  <p>We tried doing the following -</p>
  <ol>
  <li>we reduced the prefetch count (from 20 * processing rate - as mentioned in the Best Practices guide) to a bare minimum (to even <code>0</code> in one test cycle), to reduce the no. of messages that are locked for the client</li>
  <li>increased the <code>maxAutoRenewDuration</code> to 5 minutes - our <code>processDB1()</code> and <code>processDB2()</code> do not take more than a second or two for almost 90% of the cases - so, I think the lock duration of 30 seconds and <code>maxAutoRenewDuration</code> are not issues here.</li>
  <li>removed the blocking <code>CompletableFuture.get()</code> and made the processing synchronous.</li>
  </ol>
  <p>None of these tweaks helped us fix the issue. What we observed is that the <code>COMPLETE</code> or <code>RENEWMESSAGELOCK</code> are throwing the <code>MessageLockLostException</code>.</p>
  <p>We need help with finding answers for the following:</p>
  <ol>
  <li><strong>why is there a long period of inactivity of the <code>QueueClient</code> in scenario 1</strong>?</li>
  <li><strong>how do we know the <code>MessageLockLostException</code>s are thrown, because the locks have indeed expired</strong>? we suspect the locks cannot expire too soon, as our processing happens in a second or two. disabling prefetch also did not solve this for us.</li>
  </ol>
  <h2>Versions  and Service Bus details</h2>
  <ul>
  <li>Java - <code>openjdk-11-jre</code></li>
  <li>Azure Service Bus namespace tier: <code>Standard</code></li>
  <li>Java SDK version - <code>3.4.0</code></li>
  </ul>