For every new block, does the miner need to read through all historical transactions? Let's say I am about to send bob 5 bitcoins, but in the past I've done 1 million individual transactions. When a miner is putting my most recent transaction into a block, do they have to go through all 1 million transactions in order to "mine" this new block?"

I just don't understand how the miner would know how much BTC I have at the moment before this new transaction. It's not explicitly stored anywhere right?
The super naive implementation would require the miner to go through the entire blockchain. However software developers are smarter than that. Each node instead maintains a chain status. This includes it's blockchain tip (the most recent valid block), the UTXO set, and the verification status of all previous blocks. When a new block is created, the node only needs to consult its internal chain status. Using the UTXO set, it can verify whether each transaction in a block is valid. It also knows the previous blocks validity, do it does not need to go back through the blockchain and reverify those blocks.

There can be further optimizations too. Since most transactions in the block would have been unconfirmed, the node is likely to have seen them before. When it receives an unconfirmed transaction, it will verify it too. So once it receives a block, it can just check whether a transaction was one it had seen and verified before and skip its verification if it has.