@vkatsikaros (Vangelis) posted a new topic, subject "Unassigned dangling_index_imported". It read:

   I have a cluster with status "red"
    [code] 
        $ curl -s http://+1-458-718-3414:9200/_cluster/health | jq .
        {
          "active_shards_percent_as_number": 99.90479817212491,
          "task_max_waiting_in_queue_millis": 0,
          "number_of_in_flight_fetch": 0,
          "number_of_pending_tasks": 0,
          "delayed_unassigned_shards": 0,
          "unassigned_shards": 4,
          "initializing_shards": 1,
          "cluster_name": "adzunalogs",
          "status": "red",                                  <=========== oops
          "timed_out": false,
          "number_of_nodes": 3,
          "number_of_data_nodes": 3,
          "active_primary_shards": 2624,
          "active_shards": 5247,
          "relocating_shards": 1
        }
        
    [/code]
    
    and some indexes with `UNASSIGNED DANGLING_INDEX_IMPORTED`
    [code] 
        curl -s -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED
        logstash-2016.08.23 3 p UNASSIGNED DANGLING_INDEX_IMPORTED 
        logstash-2016.08.23 3 r UNASSIGNED DANGLING_INDEX_IMPORTED 
        logstash-2016.08.23 4 p UNASSIGNED DANGLING_INDEX_IMPORTED 
        logstash-2016.08.23 4 r UNASSIGNED DANGLING_INDEX_IMPORTED 
        
    [/code]
    
    All other are "green"
    [code] 
        curl -s http://+1-458-718-3414:9200/_cluster/health?level=indices | jq .
        ...
            "logstash-2016.08.23": {
              "unassigned_shards": 4,
              "initializing_shards": 0,
              "relocating_shards": 0,
              "active_shards": 6,
              "active_primary_shards": 3,
              "number_of_replicas": 1,
              "number_of_shards": 5,
              "status": "red"
            },
        ...
        
    [/code]
    
    More details on the failing indices
    [code] 
        $ curl -s -XGET localhost:9200/_cluster/health?level=shards | jq . | less
        
        
            "logstash-2016.08.23": {
              "shards": {
                "4": {
                  "unassigned_shards": 2,
                  "initializing_shards": 0,
                  "relocating_shards": 0,
                  "active_shards": 0,
                  "primary_active": false,
                  "status": "red"
                },
                "3": {
                  "unassigned_shards": 2,
                  "initializing_shards": 0,
                  "relocating_shards": 0,
                  "active_shards": 0,
                  "primary_active": false,
                  "status": "red"
                },
        
    [/code]
    [code] 
        $ curl -s 'localhost:9200/_cat/allocation?v'
        shards disk.indices disk.used disk.avail disk.total disk.percent host          ip            node       
          1749          2tb     2.3tb      1.1tb      3.5tb           66 IP1   IP1   node1   
          1749        1.8tb       2tb      1.4tb      3.5tb           58 IP2    IP2    node2
          1750        1.8tb       2tb      1.4tb      3.5tb           58 IP3    IP3 node3
             4                                                                                       UNASSIGNED 
        
        
    [/code]
    
    What would be the process of further debugging this?
    
    In https://www.elastic.co/guide/en/elasticsearch/guide/2.x/_cluster_health.html#_drilling_deeper_finding_problematic_indices I read
    
    > Once you know the index that is having problems, other APIs that we discuss in this chapter will tend to be more helpful.
    
    but I am not sure where to look at for the other APIs.
    
    How to resolve the unassigned shards says that rerouting can fix this but I am not sure if there are any implications.
    
    Edit1: fixed the quote  
    Edit2: add `health?level=shards` output

To which @vkatsikaros (Vangelis) replied

    After restarting all the cluster nodes one by one, in the end I still got those 4 unassigned shards.
    
    I followed the instructions in How to resolve the unassigned shards but failed with 400
    [code] 
        $ curl -XPOST 'http://+1-458-718-3414:9200/_cluster/reroute?pretty' -H 'Content-Type: application/json' -d'
        > {
        > "commands" : [
        > {
        > "allocate_stale_primary" : {
        > "index" : "logstash-2016.08.23",
        > "shard" : 4,
        > "node" : "my_node_name",
        > "accept_data_loss" : true
        > }
        > }
        > ]
        > }
        > '
        {
          "error" : {
            "root_cause" : [ {
              "type" : "illegal_argument_exception",
              "reason" : "No allocation command factory registered for name [allocate_stale_primary]"
            } ],
            "type" : "illegal_argument_exception",
            "reason" : "No allocation command factory registered for name [allocate_stale_primary]"
          },
          "status" : 400
        }
        
    [/code]
    
    I also tried to use the id of the node, not its node name but the result was the same, from
    [code] 
        $ curl -s http://localhost:9200/_nodes?pretty
        ...
            "3MItGKMuSXiw1YxVIVwKig" : {
              "name" : "my_node_name",
              "transport_address" : "IP:9300",
        ...
        
    [/code]

@vkatsikaros (Vangelis) replied

    I finally made ES "green" with following this https://www.datadoghq.com/blog/elasticsearch-unassigned-shards/
    [code] 
        $ curl -XPOST 'http://+1-458-718-3414:9200/_cluster/reroute' -d '
        {
            "commands" : [
               {
                   "allocate" : {
                       "index" : "logstash-2016.08.23", "shard" : 4,
                       "node": "my_node_name", "allow_primary": "true"
                   }
               }
            ]
        }
        '
        
        curl -XPOST 'http://+1-458-718-3414:9200/_cluster/reroute' -d '
        {
            "commands" : [
               {
                   "allocate" : {
                       "index" : "logstash-2016.08.23", "shard" : 3,
                       "node": "my_node_name", "allow_primary": "true"
                   }
               }
            ]
        }
        '
        
    [/code]
    
    but I was wondering if anyone can shed any light to the above.

Then @system (system) said

    This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.


{'source': 'discourse', 'url': 'https://discuss.elastic.co/t/unassigned-dangling-index-imported/97922', 'title': 'Unassigned dangling_index_imported', 'views': 8475, 'has_accepted_answer': False, 'word_count': 571, 'like_count': 0, 'reply_count': 1, 'participant_count': 1, 'posts': 4, 'tags': '', 'author': '@vkatsikaros (Vangelis)', 'participants': '@vkatsikaros (Vangelis), @system (system)', 'image_url': [None], 'created_at': '2017-08-22T14:13:34.791Z', 'last_posted_at': '2017-09-19T17:13:26.934Z'}