title: Infinispan why is a local cache waiting for a remote response
body: <p>We have a number of wildfly 18 servers running standalone but connected via message queues and infinispan caches.
  Recently we had an outage, and started seeing this error on a few of the servers :</p>
  <blockquote>
  <p>12:51:08,437 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p5-t1) ISPN000136: Error executing command RemoveExpiredCommand on Cache 'studiomanager', writing keys [20003$1860529424$187762]: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 7916397 from node4
  at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:65)
  at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
  at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_181]
  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_181]
  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_181]
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_181]
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_181]
  at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_181]</p>
  </blockquote>
  <p>Seems fairly clear except that the 'studiomanager' cache is defined as a local cache. It does make use of lifespan/eviction, which explains the first part of the error, but not the second part. Why is a local cache seemingly timing out waiting for another server as the second part of the error seems to show?</p>
  <p>We do have some replicated caches too, but this is not one of them.
  From the standalone.xml file :</p>
  <pre><code>    &lt;cache-container name=&quot;InfinispanSystem&quot; default-cache=&quot;systemSettings&quot;&gt;
          &lt;transport/&gt;
          &lt;local-cache name=&quot;studiomanager&quot;/&gt;
          &lt;replicated-cache name=&quot;systemSettings&quot;&gt;
              &lt;locking isolation=&quot;REPEATABLE_READ&quot;/&gt;
          &lt;/replicated-cache&gt;
          &lt;/snip ...&gt;
      &lt;/cache-container&gt;
  </code></pre>
  <p>I have hunted for anything that vaguely looks like the same problem, but have come up largely blank. Can anyone shed any light on how or why this could happen?</p>
  <p>EDIT:
  As requested, the infinispan version from the Wildfly log is :
  Infinispan 'Infinity Minus ONE +2' 9.4.16.Final</p>
  <p>The servers are running on docker, each using the same image so the standalone.xml config is largely the same on all of them. We do have some cli scripts that make changes to individual nodes, but the Infinispan subsystem isn't touched by them.</p>