title: AWS MSK - Brokers going out of service all of a sudden
body: <p>We had setup an MSK cluster with 3 brokers (each have 2 GB storage). The cluster was working fine for first day, then all of a sudden it stopped consuming messages on 2nd day even though there was no config change. The error we were getting at producer and consumer level was <strong>Give up sending metadata request since no node is available</strong>. Also we were not able to connect to the brokers thereafter.</p>
  <p>While troubleshooting, we came across this link- <strong><a href="https://aws.amazon.com/premiumsupport/knowledge-center/msk-cluster-connection-issues/" rel="nofollow noreferrer">https://aws.amazon.com/premiumsupport/knowledge-center/msk-cluster-connection-issues/</a></strong> . We assume, it may be due to disk utilization by looking at the cloudwatch metrics data.</p>
  <p>We want to follow best practices as shown here <strong><a href="https://docs.aws.amazon.com/msk/latest/developerguide/bestpractices.html#bestpractices-monitor-disk-space" rel="nofollow noreferrer">https://docs.aws.amazon.com/msk/latest/developerguide/bestpractices.html#bestpractices-monitor-disk-space</a></strong> but as MSK doesn't provide command line support, we are not able to flush data from brokers and also not able to restart it.</p>
  <p>It would be great if we get some advice on the same.</p>
  <p>Thanks in advance</p>