title: ElasticSearch entered "read only" mode, node cannot be altered
body: <p>Something happened during the night to my ES cluster (composed of 5 data nodes, 3 master nodes).</p>
  <p>I have no idea what happened but all the indices and data were deleted and the cluster entered a &quot;read only&quot; mode, possibly hacked?</p>
  <p>When trying to get Kibana running, I get the following:
  <a href="https://i.stack.imgur.com/l6pmY.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/l6pmY.png" alt="kibana" /></a></p>
  <p>Tried restarting Kibana - it restarted, nothing changed.
  Tried restarting Elastic - it restarted (all nodes), nothing changed.</p>
  <p>I then had a look at the cluster settings and this is what I got:</p>
  <pre><code>{
    &quot;persistent&quot;: {
      &quot;cluster&quot;: {
        &quot;routing&quot;: {
          &quot;allocation&quot;: {
            &quot;enable&quot;: &quot;all&quot;
          }
        },
        &quot;blocks&quot;: {
          &quot;read_only&quot;: &quot;true&quot;
        }
      }
    },
    &quot;transient&quot;: {
      &quot;cluster&quot;: {
        &quot;routing&quot;: {
          &quot;allocation&quot;: {
            &quot;enable&quot;: &quot;all&quot;
          }
        }
      }
    }
  }
  </code></pre>
  <p>I tried undoing the read only as follows:</p>
  <pre><code>PUT _cluster/settings
  {
    &quot;persistent&quot;: {
      &quot;blocks.read_only&quot;: false
    }
  }
  </code></pre>
  <p>No luck as you can see:</p>
  <pre><code>{
    &quot;error&quot;: {
      &quot;root_cause&quot;: [
        {
          &quot;type&quot;: &quot;cluster_block_exception&quot;,
          &quot;reason&quot;: &quot;blocked by: [FORBIDDEN/6/cluster read-only (api)];&quot;
        }
      ],
      &quot;type&quot;: &quot;cluster_block_exception&quot;,
      &quot;reason&quot;: &quot;blocked by: [FORBIDDEN/6/cluster read-only (api)];&quot;
    },
    &quot;status&quot;: 403
  }
  </code></pre>
  <p>Any ideas?</p>
  <p>UPDATE: <a href="https://stackoverflow.com/a/38736192">Problem solved by Andrei Stefan</a>, now for the more important part - why?
  What happened and why?
  I've lost all data and my cluster entered a read-only mode.</p>