Aerospike losing documents when node goes down
Question: I've been doing dome tests using aerospike and I noticed a behavior different
than what is sold.

I have a cluster of 4 nodes running on AWS in the same AZ, the instances are
t2micro (1cpu, 1gb RAM, 25gb SSD) using the aws linux with the AMI aerospike

aerospike.conf:

    
    
    heartbeat {
            mode mesh
            port 3002                        
            mesh-seed-address-port XXX.XX.XXX.164 3002
            mesh-seed-address-port XXX.XX.XXX.167 3002
            mesh-seed-address-port XXX.XX.XXX.165 3002
            #internal aws IPs
    ...
    namespace teste2 {
            replication-factor 2
            memory-size 650M
                default-ttl 365d                                                                                                                    
            storage-engine device {
                        file /opt/aerospike/data/bar.dat
                        filesize 22G
                            data-in-memory false                                                                     
            }
    }
    

What I did was a test to see if I would loose documents when a node goes down.
For that I wrote a little code on python:

    
    
    from __future__ import print_function
    import aerospike
    import pandas as pd
    import numpy as np
    import time
    import sys
    config = {
      'hosts': [ ('XX.XX.XX.XX', 3000),('XX.XX.XX.XX',3000),
                 ('XX.XX.XX.XX',3000), ('XX.XX.XX.XX',3000)]
    } # external aws ips
    client = aerospike.client(config).connect()
    for i in range(1,10000):
      key = ('teste2', 'setTest3', ''.join(('p',str(i))))
      try:
        client.put(key, {'id11': i})
        print(i)
      except Exception as e:
        print("error: {0}".format(e), file=sys.stderr)
      time.sleep(1)
    

I used this code just for inserting a sequence of integers that I could check
after that. I ran that code and after a few seconds I stopped the aerospike
service at one node for 10 seconds, using `sudo service aerospike stop` and
`sudo service aerospike colstart` to restart.

I waited for a few seconds until the nodes did all the migration and executed
the following python script:

    
    
    query = client.query('teste2', 'setTest3')
    query.select('id11')
    te = []
    def save_result((key, metadata, record)):
        te.append(record)
    query.foreach(save_result)
    d = pd.DataFrame(te)
    d2 = d.sort(columns='id11')
    te2 = np.array(d2.id11)
    for i in range(0,len(te2)):
      if i > 0:
        if (te2[i] !=  (te2[i-1]+1) ):
          print('no %d'% int(te2[i-1]+1))
    print(te2)
    

And got as response:

    
    
    no 3
    no 6
    no 8
    no 11
    no 13
    no 17
    no 20
    no 22
    no 24
    no 26
    no 30
    no 34
    no 39
    no 41
    no 48
    no 53
    [ 1  2  5  7 10 12 16 19 21 23 25 27 28 29 33 35 36 37 38 40 43 44 45 46 47 51 52 54]
    

Is my cluster configured wrong or this is normal?

ps: I tried to include as many things I could, if you please suggest more
information to include I will appreciate.


Answer: Actually I found a solution, and it is pretty simple and foolish to be honest.

In the configuration file we have some parameters for network communication
between nodes, such as:

    
    
    interval 150                    # Number of milliseconds between heartbeats
    timeout 10                      # Number of heartbeat intervals to wait
                                    # before timing out a node
    

This two parameters set the time it takes to the cluster to realize the node
is down and out of the cluster. (in this case 1.5 sec).

What we found useful was to tune the write policies at the client to work
along this parameters.

Depending on the client you will have some policies like number of tries until
the operation fails, timeout for the operation, time between tries.

You just need to adapt the client parameters. For example: set the number of
retries to 4 (each is executed after 500 ms) and the timeout to 2 sec. Doing
that the client will recognize the node is down and redirect the operation to
another node.

This setup can be overwhelming on the cluster, generating a huge overload, but
it worked for us.

