Why do my RDS read IOPS drop after a certain period of time?
I've not found the IO burst balance to work the way the documents state nor the monitoring of the burst balance to reflect reality, for t* class instances.  So while I can't explain your observation, it does not surprise me.

And in general, is there a more efficient way I can be doing this?

A single ALTER TABLE can implement a number of actions, not just a single action.  And it should generally scan the table only once, not once for each action.
ALTER TABLE table_name 
   ALTER COLUMN column_name1 SET NOT NULL,
   ALTER COLUMN column_name2 SET NOT NULL,
   ALTER COLUMN column_name3 SET NOT NULL,
   ALTER COLUMN column_name4 SET NOT NULL,
   ALTER COLUMN column_name5 SET NOT NULL...;