title: Why am I getting poor performance with Hazelcast relative to database?
body: <p>I have a cluster that I routinely run with several nodes and I am interested in resolving some performance issues. It could be that what I am doing is correct but I am not entirely sure and could use some expert guidance. The goal of this project was to offload database data into the hazelcast map to provide more scalable and performant access.</p>
  
  <p>Assume there are three nodes in the cluster and there are 30,000 entries in the container map spread roughly evenly across the cluster. For the sake of the question assume a simple structure like so with its incumbent getters, setters, constructors and so on: </p>
  
  <pre><code>class Container {
      int id;
      Set&lt;Integer&gt; dataItems;
  }
  
  class Data {
      int id;
      String value;
  }
  </code></pre>
  
  <p>The map config for the two maps looks like the following: </p>
  
  <pre><code>&lt;map name="Container"&gt;
      &lt;in-memory-format&gt;OBJECT&lt;/in-memory-format&gt;
      &lt;backup-count&gt;1&lt;/backup-count&gt;
      &lt;async-backup-count&gt;0&lt;/async-backup-count&gt;
      &lt;time-to-live-seconds&gt;0&lt;/time-to-live-seconds&gt;
      &lt;max-idle-seconds&gt;259200&lt;/max-idle-seconds&gt;
      &lt;eviction-policy&gt;LRU&lt;/eviction-policy&gt;
      &lt;max-size policy="PER_NODE"&gt;0&lt;/max-size&gt;
      &lt;eviction-percentage&gt;25&lt;/eviction-percentage&gt;
      &lt;merge-policy&gt;com.hazelcast.map.merge.PutIfAbsentMapMergePolicy&lt;/merge-policy&gt;
  &lt;/map&gt;
  </code></pre>
  
  <p>As  you can see this map has a large eviction time but is used heavily. Since the data experiences heavy write traffic as well as even heavier read traffic, I thought a near cache may not be entirely helpful as invalidations are quick. Now a standard iteration strategy if this were a local data set would be something like the following: </p>
  
  <pre><code>public List&lt;Map&lt;String, Object&gt;&gt; jsonMap(final Set&lt;Integer&gt; keys) {
      final IMap&lt;Integer, Container&gt; cmap = hazelcast.getMap("Containers");
      final IMap&lt;Integer, Data&gt; dmap = hazelcast.getMap("Data");
  
      final List&lt;Map&lt;String, Object&gt;&gt; result = new ArrayList&lt;&gt;();
      cmap.getAll(keys).values().stream().forEach((c) -&gt; {
          final Map&lt;String, Object&gt; cJson = new HashMap&lt;&gt;();
          result.add(cJson);
          cJson.put("containerId", c.id);
          final List&lt;Map&lt;String, Object&gt;&gt; dataList = new ArrayList&lt;&gt;();
          cJson.put("data", dataList);
          dmap.getAll(c.dataItems).values().stream().forEach(d -&gt; {
              final Map&lt;String, Object&gt; dJson = new HashMap&lt;&gt;();
              dataList.add(dJson);
              dJson.put("id", d.id);
              dJson.put("value", d.value);
          });
      });
      return result;
  }
  </code></pre>
  
  <p>As you can see there is simple iteration here to create a JSON representation. However since the data is scattered across the nodes we have found this to be extremely slow in performance. An order of magnitude slower than simply getting the data from the database directly. That has lead some to question the strategy of using hazelcast at all. As a solution I proposed the redesign of the system to use a completable future created with an execution callback. </p>
  
  <pre><code>public &lt;K, R&gt; CompletableFuture&lt;R&gt; submitToKeyOwner(final K key, final String executor, final Callable&lt;R&gt; callable) {
      final CompletableFuture&lt;R&gt; future = new CompletableFuture&lt;&gt;();
      hazelcast.getExecutorService(executor).submitToKeyOwner((Callable&lt;R&gt; &amp; Serializable) callable, key, new FutureExecutionCallback&lt;&gt;(future));
      return future;
  }
  
  public class FutureExecutionCallback&lt;R&gt; implements ExecutionCallback&lt;R&gt; {
      private final CompletableFuture&lt;R&gt; future;
  
      public FutureExecutionCallback(final CompletableFuture&lt;R&gt; future) {
          this.future = future;
      }
  
      @Override
      public void onResponse(final R response) {
          future.complete(response);
      }
  
      @Override
      public void onFailure(final Throwable t) {
          future.completeExceptionally(t);
      }
  }
  
  public List&lt;Map&lt;String, Object&gt;&gt; jsonMap2(final Set&lt;Integer&gt; keys) {
      final List&lt;Map&lt;String, Object&gt;&gt; result = new ArrayList&lt;&gt;();
  
      keys.stream().forEach(k -&gt; {
          result.add(submitToKeyOwner(k, (Callable&lt;Map&lt;String, Object&gt;&gt; &amp; Serializable) () -&gt; {
              final IMap&lt;Integer, Container&gt; cmap = hazelcast.getMap("Containers");
              final Container c = cmap.get(k);
              final Map&lt;String, Object&gt; cJson = new HashMap&lt;&gt;();
              cJson.put("containerId", c.id);
              final List&lt;Map&lt;String, Object&gt;&gt; dataList = new ArrayList&lt;&gt;();
              cJson.put("data", dataList);
              c.dataItems.stream().map((dk) -&gt;
                      dataList.add(submitToKeyOwner(dk, (Callable&lt;Map&lt;String, Object&gt;&gt; &amp; Serializable) () -&gt; {
                          final IMap&lt;Integer, Data&gt; dmap = hazelcast.getMap("Data");
                          final Data d = dmap.get(dk);
                          final Map&lt;String, Object&gt; dJson = new HashMap&lt;&gt;();
                          dJson.put("id", d.id);
                          dJson.put("value", d.value);
                          return dJson;
                      }).join()));
              return cJson;
          }).join());
      });
  
      return result;
  }
  </code></pre>
  
  <p>Essentially I have devolved everything into a submitToKey and used Completable futures to wrap it all up. The logic being that the fetch of the object will run only on the node where it is locally stored. Although this works it is still slower than accessing the database directly given the hundreds of records we are accessing when a single database Hibernate call would bring the records in nanoseconds, this one is measured in tens of milliseconds. That seems counterintuitive in some ways.  I would think the access to the cache should be much quicker than it actually is. Perhaps I am doing something wrong both in the implementation of the iteration or just a general paradigm. Entry processors are not an option because although I have posted a trivial example, the real example uses other maps in its process as well and entry processors have serious limitations. Using map reduce is not appropriate because the administration overhead of the job has proven to be more costly than either of these two methods.</p>
  
  <p>The question I have is whether each of these is the right paradigm and if I should be expecting tens or hundreds of milliseconds in latency? Is this just the cost of doing business in a clusterable fault tolerant world or is there something I can do to reduce the time? Finally is there any better paradigm to use when accessing data in this manner? </p>
  
  <p>Thanks a bunch for your time.</p>