title: Hazelcast and MapDB - implementing a simple distributed database
body: <p>I have implemented a hazelcast service which stores its data into local mapdb instances via MapStoreFactory and newMapLoader. This way the keys can be loaded if a cluster restart is necessary:</p>
  
  <pre><code>public class HCMapStore&lt;V&gt; implements MapStore&lt;String, V&gt; {
  
  Map&lt;String, V&gt; map;
  
  /** specify the mapdb e.g. via 
    * DBMaker.newFileDB(new File("mapdb")).closeOnJvmShutdown().make() 
    */
  public HCMapStore(DB db) {
      this.db = db;
      this.map = db.createHashMap("someMapName").&lt;String, Object&gt;makeOrGet();
  }
  
  // some other store methods are omitted
  @Override
  public void delete(String k) {
      logger.info("delete, " + k);
      map.remove(k);
      db.commit();
  }
  
  // MapLoader methods
  @Override
  public V load(String key) {
      logger.info("load, " + key);
      return map.get(key);
  }
  
  @Override
  public Set&lt;String&gt; loadAllKeys() {
      logger.info("loadAllKeys");
      return map.keySet();
  }
  
  @Override
  public Map&lt;String, V&gt; loadAll(Collection&lt;String&gt; keys) {
      logger.info("loadAll, " + keys);
      Map&lt;String, V&gt; partialMap = new HashMap&lt;&gt;();
      for (String k : keys) {
          partialMap.put(k, map.get(k));
      }
      return partialMap;
  }}
  </code></pre>
  
  <p>The problem I'm now facing is that the loadAllKeys method of the MapLoader interface from hazelcast requires to return ALL keys of the whole cluster BUT every node stores ONLY the objects it owns.</p>
  
  <p>Example: I have two nodes and store 8 objects, then e.g. 5 objects are stored in the mapdb of node1 and 3 in the mapdb of node2. Which object is owned by which node is decided by hazelcast I think. Now on restart node1 will return 5 keys for loadAllKeys and node2 will return 3. Hazelcast decides to ignore the 3 items and data is 'lost'. </p>
  
  <p><strong>What could be a good solution to this?</strong></p>
  
  <p><strong>Update for bounty</strong>: <a href="https://groups.google.com/forum/#!topic/hazelcast/t91IToUX3AE" rel="nofollow">Here</a> I asked this on the hc mailing list mentioning 2 options (I'll add 1 more) and I would like to know if something like this is already possible with hazelcast 3.2 or 3.3:</p>
  
  <ol>
  <li><p>Currently the MapStore interface gets only data or updates from the local node. Would it be possible to notify the MapStore interface of every storage action of the full cluster? Or maybe this is already possible with some listener magic? Maybe I can force hazelcast to put all objects into one partition and have 1 copy on every node.</p></li>
  <li><p>If I restart e.g. 2 nodes then the MapStore interface gets called correctly with my local databases for node1 and then for node2. But when both nodes join the data of node2 will be removed as Hazelcast assumes that only the master node can be correct. Could I teach hazelcast to accept the data from both nodes?</p></li>
  </ol>