title: Amazon RDS PostgreSQL: Sudden increase in Read IOPS
body: <p>We are using Amazon RDS to host our PostgreSQL databases. Our production instance (db.t3.xlarge, Single-AZ) was running smoothly until suddenly <code>Read IOPS</code>, <code>Read Latency</code>, <code>Read Throughput</code> and <code>Disk Queue Depth</code> metrics in the AWS console increased rapidly and stayed high afterward (with a lower variability) whereas <code>Write IOPS</code> and <code>Write Throughput</code> were normal.</p>
  <p><a href="https://i.stack.imgur.com/Lwlmt.png" rel="nofollow noreferrer">Read IOPS</a></p>
  <p><a href="https://i.stack.imgur.com/LJZUP.png" rel="nofollow noreferrer">Read Throughput</a></p>
  <p><a href="https://i.stack.imgur.com/STWxT.png" rel="nofollow noreferrer">Disk Queue Depth</a></p>
  <p><a href="https://i.stack.imgur.com/KK6Sk.png" rel="nofollow noreferrer">Write IOPS</a></p>
  <p>There were no code changes or deployments on the date of the increase. There were no significant increases in user activity either.</p>
  <p>About our DB structure, we have a single table that holds all of our data and in that table, we have these fields: <code>id</code> as UUID (primary key), <code>type</code> as VARCHAR, <code>data</code> as JSONB (holds the actual data), <code>createdAt</code> and <code>updatedAt</code> as timestamp with the time zone. Most of our data columns have sizes bigger than 2 KB so most of the rows are stored in TOAST table. We have 20 (BTREE) indexes that are created for frequently used fields in JSONB.</p>
  <p>So far we have tried <code>VACUUM ANALYZE</code> and also completely rebuilding our table: creating a new table, copying all data from the old table, creating all indexes. They didn't change the behavior.</p>
  <p>We also tried increasing storage thus increasing IOPS performance. It helped a bit but it is still not the same as before.</p>
  <p>What could be the root cause of this problem? How can we fix it permanently (without increasing storage or instance type)? For now, we are looking for easy changes and we will improve our data model in the future.</p>