title: Mosquitto MQTT Disk Read Spike
body: <p>I've got an issue with one of our MQTT brokers running Mosquitto. Where there is a consistent increase in disk usage at a seemingly random time, until it reaches an equilibrium of about 50MB/s read (Shown in graph bellow).</p>
  <p>This also causes a secondary issue with our PHP server (Laravel, PHP:7.4 running php-mqtt/client: v1.1.0), where once a connection issue occurs PHP CPU spikes to 100% until the broker is restarted and then PHP server is restarted (From testing must be in that order, else problem still persists). MQTT messages are broadcast on all Laravel jobs as part of a queue monitoring system. A note during this PHP issues, connections to the broker slowly increase, I assume this is just due to the broker holding them open but too slow to process them.</p>
  <p>At a guess it might be something like QoS &gt;0 messages stacking up on the broker causing excessive disk usage. Though I can't confirm this and would expect high disk write instead of read. I've checked for network bottlenecks and other potential causes but nothing found so far.</p>
  <p>Currently at a loss as to what could cause this issue as 99% of the time the broker operates normally and there is nothing we know of that would trigger this consistent increase in disk I/O. Issue also occurred several weeks ago but at a different time of day.</p>
  <h4>Broker is running on a GCE Instance with the following specs:</h4>
  <ul>
  <li>CPU: 2 vCPUs (E2 : Intel Broadwell)</li>
  <li>Memory: 4 GB</li>
  <li>Disk: 250GB (Max Throughput: 70MB/s)</li>
  <li>OS: Container-Optimized OS, 89</li>
  </ul>
  <h3>Broker disk monitoring</h3>
  <p>Normal throughput is what is seen before 1:30am
  <a href="https://i.stack.imgur.com/th4pI.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/th4pI.png" alt="Mosquitto Disk Throughput" /></a></p>
  <h3>Broker network traffic</h3>
  <p>Network traffic is within the norms for this time of day. Dropped between 5:30 and 6:30 due to us trying to fix the issue.
  <a href="https://i.stack.imgur.com/41Dae.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/41Dae.png" alt="enter image description here" /></a></p>
  <h3>Broker CPU</h3>
  <p>Never seems to reach 100% but could be due to high frequency spikes that the monitoring is averaging.
  <a href="https://i.stack.imgur.com/EALbZ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/EALbZ.png" alt="enter image description here" /></a></p>
  <h3>PHP Server CPU</h3>
  <p>Note CPU does not spike for serval hours after broker shows signs of a problem.
  <a href="https://i.stack.imgur.com/OQR9G.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/OQR9G.png" alt="enter image description here" /></a></p>
  <h3>Mosquitto Version</h3>
  <p>Official Docker Container: <code>eclipse-mosquitto:1.6.14</code></p>
  <h3>Mosquitto Config</h3>
  <pre><code>listener 1883 0.0.0.0
  protocol mqtt
  
  listener 9001 0.0.0.0
  protocol websockets
  
  allow_anonymous false
  password_file /mosquitto/config/mosquitto.passwd
  acl_file /mosquitto/config/mosquitto.acl
  </code></pre>
  <h3>Normal Traffic</h3>
  <p>Messages: ~1000 per minute throughput on the broker</p>
  <p>PHP server only publishes short-lived messages with the following config:</p>
  <pre><code>QoS: 1
  Retain: 0
  Timeout: 1
  </code></pre>
  <h3>Dump of some $SYS topics as of now:</h3>
  <pre><code>$SYS/broker/version: &quot;mosquitto version 1.6.14&quot;
  $SYS/broker/uptime: &quot;23529 seconds&quot;
  $SYS/broker/messages/stored: &quot;93&quot; // Remained stable since monitoring
  $SYS/broker/messages/received: &quot;895858&quot;
  $SYS/broker/messages/sent: &quot;1405546&quot;
  $SYS/broker/retained messages/count: &quot;93&quot; // Remained stable since monitoring
  $SYS/broker/clients/total: &quot;122078&quot;
  $SYS/broker/clients/inactive: &quot;121984&quot;
  $SYS/broker/clients/disconnected: &quot;121984&quot;
  $SYS/broker/clients/active: &quot;94&quot;
  $SYS/broker/clients/connected: &quot;94&quot;
  $SYS/broker/clients/maximum: &quot;122078&quot;
  $SYS/broker/subscriptions/count: &quot;393&quot;
  </code></pre>
  <p>In the process of adding monitoring to these, gather data if the issue occurs again.</p>