Do miners keep a "balance" of how much each public key has?
The notion of accounts is for human convenience. The system doesn't care that you think of all the coins accessible by the same private key as a single account.

Instead, miners keep an efficiently organized database of all UTXOs (unspent transaction outputs). When a miner validates a transaction, it checks that each input corresponds to a valid UTXO.

There's no need to keep the UTXO database in memory. Storing it on disk is fine as long as it is efficiently indexed. Since it's kept for precisely for this purpose, it is efficiently indexed for this purpose.