Adding columns upped my RAM usage significantly
How many GB in the big table?

What was the setting of innodb_file_per_table?  Of innodb_buffer_pool_size?  Please provide the ALTER statement(s).

What probably happened was that the table was copied over to produce the new version with the new columns.  (8.0 can add columns without copying -- in some situations.)

If you had file_per_table OFF, then the copy would be in ibdata1, which won't shrink afterwards.  (The freed up space will be reused, just not returned to the OS, which you are monitoring.)

The buffer_pool is in RAM.  It grows as needed up to the configured limit, which I guess to be 5G.  The ALTER caused the sudden growth.  This is a "cache", so there is no 'permanent' harm.  Again, no shrinkage will happen.