How can a 24 word seed know when I added or removed wallets at a later point?
I don't know about Ledger specifically, but for HD wallets in general, private and public keys are not generated randomly.  Rather, they are generated by a deterministic algorithm from the seed (in this case, the 24 words).  So they are absolutely "tied" to the seed.

When you add a wallet for another coin, keys for that wallet would presumably be generated from the same seed (using different parameters so that you don't end up with the same keys in all your wallets).    The seed itself stays the same and doesn't have to change to reflect the creation of the new wallet.

Should you ever need to restore, I assume you would need to tell the device which coins you had wallets for.  It will then regenerate all the keys for those wallets from the master seed, using the same algorithm as it used originally, and you'll have all the same keys you had before.  (If you try to restore some wallets that you weren't using or had removed, all that will happen is you'll find no coins in that wallet.)

Your public and private keys are not stored anywhere except within the device itself.