<p dir="auto">We take snpashots every hour to AWS S3 directly using cloud-aws plugin<br>
snapshot was stuck because of a reboot on master node during the snapshot, ES version 1.4.0, ArchLinux<br>
Cluster was later upgraded to 1.4.2 at which point we noticed that there is a snapshot stuck<br>
while trying to create a new snapshot<br>
<code class="notranslate">{"error":"ConcurrentSnapshotExecutionException[[my_s3_repository:2015.04.13.09.16.42] a snapshot is already running]","status":503}</code><br>
curl localhost:9200/_snapshot/my_s3_repository/_all?pretty doesn't have the stuck snapshot in the list.</p>
<p dir="auto">Delete snapshot command never completes, it hangs forever.<br>
rolling restart (restarting one node after the other) didn't help.<br>
Clean up script doesn't work for me <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="66571776" data-permission-text="Title is private" data-url="https://github.com/imotov/elasticsearch-snapshot-cleanup/issues/2" data-hovercard-type="issue" data-hovercard-url="/imotov/elasticsearch-snapshot-cleanup/issues/2/hovercard" href="https://github.com/imotov/elasticsearch-snapshot-cleanup/issues/2">imotov/elasticsearch-snapshot-cleanup#2</a></p>
<p dir="auto">In S3, there is no metadata file for the stuck snapshot, there are snapshot- files for some of the indices</p>
<p dir="auto">How do I atleast forget/discard old snapshots..create a new repository and start fresh. I can not stop the whole cluster as it is a production cluster.</p>
<p dir="auto">curl localhost:9200/_snapshot/_status?pretty</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
  &quot;snapshots&quot; : [ {
    &quot;snapshot&quot; : &quot;2015.03.11.06.04.39&quot;,
    &quot;repository&quot; : &quot;my_s3_repository&quot;,
    &quot;state&quot; : &quot;ABORTED&quot;,
    &quot;shards_stats&quot; : {
      &quot;initializing&quot; : 0,
      &quot;started&quot; : 0,
      &quot;finalizing&quot; : 0,
      &quot;done&quot; : 17,
      &quot;failed&quot; : 3,
      &quot;total&quot; : 20
    },
    &quot;stats&quot; : {
      &quot;number_of_files&quot; : 0,
      &quot;processed_files&quot; : 0,
      &quot;total_size_in_bytes&quot; : 0,
      &quot;processed_size_in_bytes&quot; : 0,
      &quot;start_time_in_millis&quot; : 0,
      &quot;time_in_millis&quot; : 0
    },
    &quot;indices&quot; : {
      &quot;playground_video&quot; : {
        &quot;shards_stats&quot; : {
          &quot;initializing&quot; : 0,
          &quot;started&quot; : 0,
          &quot;finalizing&quot; : 0,
          &quot;done&quot; : 5,
          &quot;failed&quot; : 0,
          &quot;total&quot; : 5
        },
        &quot;stats&quot; : {
          &quot;number_of_files&quot; : 0,
          &quot;processed_files&quot; : 0,
          &quot;total_size_in_bytes&quot; : 0,
          &quot;processed_size_in_bytes&quot; : 0,
          &quot;start_time_in_millis&quot; : 0,
          &quot;time_in_millis&quot; : 0
        },
        &quot;shards&quot; : {
          &quot;4&quot; : {
            &quot;stage&quot; : &quot;DONE&quot;,
            &quot;stats&quot; : {
              &quot;number_of_files&quot; : 0,
              &quot;processed_files&quot; : 0,
              &quot;total_size_in_bytes&quot; : 0,
              &quot;processed_size_in_bytes&quot; : 0,
              &quot;start_time_in_millis&quot; : 0,
              &quot;time_in_millis&quot; : 0
            }
          },
          &quot;3&quot; : {
            &quot;stage&quot; : &quot;DONE&quot;,
            &quot;stats&quot; : {
              &quot;number_of_files&quot; : 0,
              &quot;processed_files&quot; : 0,
              &quot;total_size_in_bytes&quot; : 0,
              &quot;processed_size_in_bytes&quot; : 0,
              &quot;start_time_in_millis&quot; : 0,
              &quot;time_in_millis&quot; : 0
            }
          },
          &quot;2&quot; : {
            &quot;stage&quot; : &quot;DONE&quot;,
            &quot;stats&quot; : {
              &quot;number_of_files&quot; : 0,
              &quot;processed_files&quot; : 0,
              &quot;total_size_in_bytes&quot; : 0,
              &quot;processed_size_in_bytes&quot; : 0,
              &quot;start_time_in_millis&quot; : 0,
              &quot;time_in_millis&quot; : 0
            }
          },
          &quot;1&quot; : {
            &quot;stage&quot; : &quot;DONE&quot;,
            &quot;stats&quot; : {
              &quot;number_of_files&quot; : 0,
              &quot;processed_files&quot; : 0,
              &quot;total_size_in_bytes&quot; : 0,
              &quot;processed_size_in_bytes&quot; : 0,
              &quot;start_time_in_millis&quot; : 0,
              &quot;time_in_millis&quot; : 0
            }
          },
          &quot;0&quot; : {
            &quot;stage&quot; : &quot;DONE&quot;,
            &quot;stats&quot; : {
              &quot;number_of_files&quot; : 0,
              &quot;processed_files&quot; : 0,
              &quot;total_size_in_bytes&quot; : 0,
              &quot;processed_size_in_bytes&quot; : 0,
              &quot;start_time_in_millis&quot; : 0,
              &quot;time_in_millis&quot; : 0
            }
          }
        }
      },
      &quot;playground_feed&quot; : {
        &quot;shards_stats&quot; : {
          &quot;initializing&quot; : 0,
          &quot;started&quot; : 0,
          &quot;finalizing&quot; : 0,
          &quot;done&quot; : 4,
          &quot;failed&quot; : 1,
          &quot;total&quot; : 5
        },
        &quot;stats&quot; : {
          &quot;number_of_files&quot; : 0,
          &quot;processed_files&quot; : 0,
          &quot;total_size_in_bytes&quot; : 0,
          &quot;processed_size_in_bytes&quot; : 0,
          &quot;start_time_in_millis&quot; : 0,
          &quot;time_in_millis&quot; : 0
        },
        &quot;shards&quot; : {
          &quot;4&quot; : {
            &quot;stage&quot; : &quot;DONE&quot;,
            &quot;stats&quot; : {
              &quot;number_of_files&quot; : 0,
              &quot;processed_files&quot; : 0,
              &quot;total_size_in_bytes&quot; : 0,
              &quot;processed_size_in_bytes&quot; : 0,
              &quot;start_time_in_millis&quot; : 0,
              &quot;time_in_millis&quot; : 0
            }
          },
          &quot;2&quot; : {
            &quot;stage&quot; : &quot;DONE&quot;,
            &quot;stats&quot; : {
              &quot;number_of_files&quot; : 0,
              &quot;processed_files&quot; : 0,
              &quot;total_size_in_bytes&quot; : 0,
              &quot;processed_size_in_bytes&quot; : 0,
              &quot;start_time_in_millis&quot; : 0,
              &quot;time_in_millis&quot; : 0
            }
          },
          &quot;3&quot; : {
            &quot;stage&quot; : &quot;DONE&quot;,
            &quot;stats&quot; : {
              &quot;number_of_files&quot; : 0,
              &quot;processed_files&quot; : 0,
              &quot;total_size_in_bytes&quot; : 0,
              &quot;processed_size_in_bytes&quot; : 0,
              &quot;start_time_in_millis&quot; : 0,
              &quot;time_in_millis&quot; : 0
            }
          },
          &quot;0&quot; : {
            &quot;stage&quot; : &quot;DONE&quot;,
            &quot;stats&quot; : {
              &quot;number_of_files&quot; : 0,
              &quot;processed_files&quot; : 0,
              &quot;total_size_in_bytes&quot; : 0,
              &quot;processed_size_in_bytes&quot; : 0,
              &quot;start_time_in_millis&quot; : 0,
              &quot;time_in_millis&quot; : 0
            }
          },
          &quot;1&quot; : {
            &quot;stage&quot; : &quot;FAILURE&quot;,
            &quot;stats&quot; : {
              &quot;number_of_files&quot; : 0,
              &quot;processed_files&quot; : 0,
              &quot;total_size_in_bytes&quot; : 0,
              &quot;processed_size_in_bytes&quot; : 0,
              &quot;start_time_in_millis&quot; : 0,
              &quot;time_in_millis&quot; : 0
            }
          }
        }
      },
      &quot;playground_profile&quot; : {
        &quot;shards_stats&quot; : {
          &quot;initializing&quot; : 0,
          &quot;started&quot; : 0,
          &quot;finalizing&quot; : 0,
          &quot;done&quot; : 4,
          &quot;failed&quot; : 1,
          &quot;total&quot; : 5
        },
        &quot;stats&quot; : {
          &quot;number_of_files&quot; : 0,
          &quot;processed_files&quot; : 0,
          &quot;total_size_in_bytes&quot; : 0,
          &quot;processed_size_in_bytes&quot; : 0,
          &quot;start_time_in_millis&quot; : 0,
          &quot;time_in_millis&quot; : 0
        },
        &quot;shards&quot; : {
          &quot;2&quot; : {
            &quot;stage&quot; : &quot;DONE&quot;,
            &quot;stats&quot; : {
              &quot;number_of_files&quot; : 0,
              &quot;processed_files&quot; : 0,
              &quot;total_size_in_bytes&quot; : 0,
              &quot;processed_size_in_bytes&quot; : 0,
              &quot;start_time_in_millis&quot; : 0,
              &quot;time_in_millis&quot; : 0
            }
          },
          &quot;1&quot; : {
            &quot;stage&quot; : &quot;DONE&quot;,
            &quot;stats&quot; : {
              &quot;number_of_files&quot; : 0,
              &quot;processed_files&quot; : 0,
              &quot;total_size_in_bytes&quot; : 0,
              &quot;processed_size_in_bytes&quot; : 0,
              &quot;start_time_in_millis&quot; : 0,
              &quot;time_in_millis&quot; : 0
            }
          },
          &quot;4&quot; : {
            &quot;stage&quot; : &quot;DONE&quot;,
            &quot;stats&quot; : {
              &quot;number_of_files&quot; : 0,
              &quot;processed_files&quot; : 0,
              &quot;total_size_in_bytes&quot; : 0,
              &quot;processed_size_in_bytes&quot; : 0,
              &quot;start_time_in_millis&quot; : 0,
              &quot;time_in_millis&quot; : 0
            }
          },
          &quot;3&quot; : {
            &quot;stage&quot; : &quot;DONE&quot;,
            &quot;stats&quot; : {
              &quot;number_of_files&quot; : 0,
              &quot;processed_files&quot; : 0,
              &quot;total_size_in_bytes&quot; : 0,
              &quot;processed_size_in_bytes&quot; : 0,
              &quot;start_time_in_millis&quot; : 0,
              &quot;time_in_millis&quot; : 0
            }
          },
          &quot;0&quot; : {
            &quot;stage&quot; : &quot;FAILURE&quot;,
            &quot;stats&quot; : {
              &quot;number_of_files&quot; : 0,
              &quot;processed_files&quot; : 0,
              &quot;total_size_in_bytes&quot; : 0,
              &quot;processed_size_in_bytes&quot; : 0,
              &quot;start_time_in_millis&quot; : 0,
              &quot;time_in_millis&quot; : 0
            }
          }
        }
      },
      &quot;playground_admin&quot; : {
        &quot;shards_stats&quot; : {
          &quot;initializing&quot; : 0,
          &quot;started&quot; : 0,
          &quot;finalizing&quot; : 0,
          &quot;done&quot; : 4,
          &quot;failed&quot; : 1,
          &quot;total&quot; : 5
        },
        &quot;stats&quot; : {
          &quot;number_of_files&quot; : 0,
          &quot;processed_files&quot; : 0,
          &quot;total_size_in_bytes&quot; : 0,
          &quot;processed_size_in_bytes&quot; : 0,
          &quot;start_time_in_millis&quot; : 0,
          &quot;time_in_millis&quot; : 0
        },
        &quot;shards&quot; : {
          &quot;2&quot; : {
            &quot;stage&quot; : &quot;DONE&quot;,
            &quot;stats&quot; : {
              &quot;number_of_files&quot; : 0,
              &quot;processed_files&quot; : 0,
              &quot;total_size_in_bytes&quot; : 0,
              &quot;processed_size_in_bytes&quot; : 0,
              &quot;start_time_in_millis&quot; : 0,
              &quot;time_in_millis&quot; : 0
            }
          },
          &quot;1&quot; : {
            &quot;stage&quot; : &quot;DONE&quot;,
            &quot;stats&quot; : {
              &quot;number_of_files&quot; : 0,
              &quot;processed_files&quot; : 0,
              &quot;total_size_in_bytes&quot; : 0,
              &quot;processed_size_in_bytes&quot; : 0,
              &quot;start_time_in_millis&quot; : 0,
              &quot;time_in_millis&quot; : 0
            }
          },
          &quot;0&quot; : {
            &quot;stage&quot; : &quot;DONE&quot;,
            &quot;stats&quot; : {
              &quot;number_of_files&quot; : 0,
              &quot;processed_files&quot; : 0,
              &quot;total_size_in_bytes&quot; : 0,
              &quot;processed_size_in_bytes&quot; : 0,
              &quot;start_time_in_millis&quot; : 0,
              &quot;time_in_millis&quot; : 0
            }
          },
          &quot;4&quot; : {
            &quot;stage&quot; : &quot;FAILURE&quot;,
            &quot;stats&quot; : {
              &quot;number_of_files&quot; : 0,
              &quot;processed_files&quot; : 0,
              &quot;total_size_in_bytes&quot; : 0,
              &quot;processed_size_in_bytes&quot; : 0,
              &quot;start_time_in_millis&quot; : 0,
              &quot;time_in_millis&quot; : 0
            }
          },
          &quot;3&quot; : {
            &quot;stage&quot; : &quot;DONE&quot;,
            &quot;stats&quot; : {
              &quot;number_of_files&quot; : 0,
              &quot;processed_files&quot; : 0,
              &quot;total_size_in_bytes&quot; : 0,
              &quot;processed_size_in_bytes&quot; : 0,
              &quot;start_time_in_millis&quot; : 0,
              &quot;time_in_millis&quot; : 0
            }
          }
        }
      }
    }
  } ]
}"><pre class="notranslate"><code class="notranslate">{
  "snapshots" : [ {
    "snapshot" : "2015.03.11.06.04.39",
    "repository" : "my_s3_repository",
    "state" : "ABORTED",
    "shards_stats" : {
      "initializing" : 0,
      "started" : 0,
      "finalizing" : 0,
      "done" : 17,
      "failed" : 3,
      "total" : 20
    },
    "stats" : {
      "number_of_files" : 0,
      "processed_files" : 0,
      "total_size_in_bytes" : 0,
      "processed_size_in_bytes" : 0,
      "start_time_in_millis" : 0,
      "time_in_millis" : 0
    },
    "indices" : {
      "playground_video" : {
        "shards_stats" : {
          "initializing" : 0,
          "started" : 0,
          "finalizing" : 0,
          "done" : 5,
          "failed" : 0,
          "total" : 5
        },
        "stats" : {
          "number_of_files" : 0,
          "processed_files" : 0,
          "total_size_in_bytes" : 0,
          "processed_size_in_bytes" : 0,
          "start_time_in_millis" : 0,
          "time_in_millis" : 0
        },
        "shards" : {
          "4" : {
            "stage" : "DONE",
            "stats" : {
              "number_of_files" : 0,
              "processed_files" : 0,
              "total_size_in_bytes" : 0,
              "processed_size_in_bytes" : 0,
              "start_time_in_millis" : 0,
              "time_in_millis" : 0
            }
          },
          "3" : {
            "stage" : "DONE",
            "stats" : {
              "number_of_files" : 0,
              "processed_files" : 0,
              "total_size_in_bytes" : 0,
              "processed_size_in_bytes" : 0,
              "start_time_in_millis" : 0,
              "time_in_millis" : 0
            }
          },
          "2" : {
            "stage" : "DONE",
            "stats" : {
              "number_of_files" : 0,
              "processed_files" : 0,
              "total_size_in_bytes" : 0,
              "processed_size_in_bytes" : 0,
              "start_time_in_millis" : 0,
              "time_in_millis" : 0
            }
          },
          "1" : {
            "stage" : "DONE",
            "stats" : {
              "number_of_files" : 0,
              "processed_files" : 0,
              "total_size_in_bytes" : 0,
              "processed_size_in_bytes" : 0,
              "start_time_in_millis" : 0,
              "time_in_millis" : 0
            }
          },
          "0" : {
            "stage" : "DONE",
            "stats" : {
              "number_of_files" : 0,
              "processed_files" : 0,
              "total_size_in_bytes" : 0,
              "processed_size_in_bytes" : 0,
              "start_time_in_millis" : 0,
              "time_in_millis" : 0
            }
          }
        }
      },
      "playground_feed" : {
        "shards_stats" : {
          "initializing" : 0,
          "started" : 0,
          "finalizing" : 0,
          "done" : 4,
          "failed" : 1,
          "total" : 5
        },
        "stats" : {
          "number_of_files" : 0,
          "processed_files" : 0,
          "total_size_in_bytes" : 0,
          "processed_size_in_bytes" : 0,
          "start_time_in_millis" : 0,
          "time_in_millis" : 0
        },
        "shards" : {
          "4" : {
            "stage" : "DONE",
            "stats" : {
              "number_of_files" : 0,
              "processed_files" : 0,
              "total_size_in_bytes" : 0,
              "processed_size_in_bytes" : 0,
              "start_time_in_millis" : 0,
              "time_in_millis" : 0
            }
          },
          "2" : {
            "stage" : "DONE",
            "stats" : {
              "number_of_files" : 0,
              "processed_files" : 0,
              "total_size_in_bytes" : 0,
              "processed_size_in_bytes" : 0,
              "start_time_in_millis" : 0,
              "time_in_millis" : 0
            }
          },
          "3" : {
            "stage" : "DONE",
            "stats" : {
              "number_of_files" : 0,
              "processed_files" : 0,
              "total_size_in_bytes" : 0,
              "processed_size_in_bytes" : 0,
              "start_time_in_millis" : 0,
              "time_in_millis" : 0
            }
          },
          "0" : {
            "stage" : "DONE",
            "stats" : {
              "number_of_files" : 0,
              "processed_files" : 0,
              "total_size_in_bytes" : 0,
              "processed_size_in_bytes" : 0,
              "start_time_in_millis" : 0,
              "time_in_millis" : 0
            }
          },
          "1" : {
            "stage" : "FAILURE",
            "stats" : {
              "number_of_files" : 0,
              "processed_files" : 0,
              "total_size_in_bytes" : 0,
              "processed_size_in_bytes" : 0,
              "start_time_in_millis" : 0,
              "time_in_millis" : 0
            }
          }
        }
      },
      "playground_profile" : {
        "shards_stats" : {
          "initializing" : 0,
          "started" : 0,
          "finalizing" : 0,
          "done" : 4,
          "failed" : 1,
          "total" : 5
        },
        "stats" : {
          "number_of_files" : 0,
          "processed_files" : 0,
          "total_size_in_bytes" : 0,
          "processed_size_in_bytes" : 0,
          "start_time_in_millis" : 0,
          "time_in_millis" : 0
        },
        "shards" : {
          "2" : {
            "stage" : "DONE",
            "stats" : {
              "number_of_files" : 0,
              "processed_files" : 0,
              "total_size_in_bytes" : 0,
              "processed_size_in_bytes" : 0,
              "start_time_in_millis" : 0,
              "time_in_millis" : 0
            }
          },
          "1" : {
            "stage" : "DONE",
            "stats" : {
              "number_of_files" : 0,
              "processed_files" : 0,
              "total_size_in_bytes" : 0,
              "processed_size_in_bytes" : 0,
              "start_time_in_millis" : 0,
              "time_in_millis" : 0
            }
          },
          "4" : {
            "stage" : "DONE",
            "stats" : {
              "number_of_files" : 0,
              "processed_files" : 0,
              "total_size_in_bytes" : 0,
              "processed_size_in_bytes" : 0,
              "start_time_in_millis" : 0,
              "time_in_millis" : 0
            }
          },
          "3" : {
            "stage" : "DONE",
            "stats" : {
              "number_of_files" : 0,
              "processed_files" : 0,
              "total_size_in_bytes" : 0,
              "processed_size_in_bytes" : 0,
              "start_time_in_millis" : 0,
              "time_in_millis" : 0
            }
          },
          "0" : {
            "stage" : "FAILURE",
            "stats" : {
              "number_of_files" : 0,
              "processed_files" : 0,
              "total_size_in_bytes" : 0,
              "processed_size_in_bytes" : 0,
              "start_time_in_millis" : 0,
              "time_in_millis" : 0
            }
          }
        }
      },
      "playground_admin" : {
        "shards_stats" : {
          "initializing" : 0,
          "started" : 0,
          "finalizing" : 0,
          "done" : 4,
          "failed" : 1,
          "total" : 5
        },
        "stats" : {
          "number_of_files" : 0,
          "processed_files" : 0,
          "total_size_in_bytes" : 0,
          "processed_size_in_bytes" : 0,
          "start_time_in_millis" : 0,
          "time_in_millis" : 0
        },
        "shards" : {
          "2" : {
            "stage" : "DONE",
            "stats" : {
              "number_of_files" : 0,
              "processed_files" : 0,
              "total_size_in_bytes" : 0,
              "processed_size_in_bytes" : 0,
              "start_time_in_millis" : 0,
              "time_in_millis" : 0
            }
          },
          "1" : {
            "stage" : "DONE",
            "stats" : {
              "number_of_files" : 0,
              "processed_files" : 0,
              "total_size_in_bytes" : 0,
              "processed_size_in_bytes" : 0,
              "start_time_in_millis" : 0,
              "time_in_millis" : 0
            }
          },
          "0" : {
            "stage" : "DONE",
            "stats" : {
              "number_of_files" : 0,
              "processed_files" : 0,
              "total_size_in_bytes" : 0,
              "processed_size_in_bytes" : 0,
              "start_time_in_millis" : 0,
              "time_in_millis" : 0
            }
          },
          "4" : {
            "stage" : "FAILURE",
            "stats" : {
              "number_of_files" : 0,
              "processed_files" : 0,
              "total_size_in_bytes" : 0,
              "processed_size_in_bytes" : 0,
              "start_time_in_millis" : 0,
              "time_in_millis" : 0
            }
          },
          "3" : {
            "stage" : "DONE",
            "stats" : {
              "number_of_files" : 0,
              "processed_files" : 0,
              "total_size_in_bytes" : 0,
              "processed_size_in_bytes" : 0,
              "start_time_in_millis" : 0,
              "time_in_millis" : 0
            }
          }
        }
      }
    }
  } ]
}
</code></pre></div>
<p dir="auto">The issue was originally reported in <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="44824333" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/7980" data-hovercard-type="issue" data-hovercard-url="/elastic/elasticsearch/issues/7980/hovercard?comment_id=76151889&amp;comment_type=issue_comment" href="https://github.com/elastic/elasticsearch/issues/7980#issuecomment-76151889">#7980 (comment)</a> If a current master node that contains all primary shards is restarted in the middle of snapshot operation, it might leave the snapshot hanging in <code class="notranslate">ABORTED</code> state.</p>