Why isn't the block hash used to calculate accumulated work? Why is this idea flawed? It doesn't seem to be a bad idea at first, because the miners should be incentivized to extend existing chain, instead of to "remine existing blocks", since such "remining" apparently requires more work to be done.

What's the actual reason? Because it would make double spending easier? Or it would enable miners to do selfish mining when they occasionally find a block with extra-small hash? Or it would cause more forks / require more confirmations?

I don't think this problem is duplicate with another question, because difficulty decides validity of a single block, while accumulated work decides best chain.
The answer is because among the set of acceptable hashes, every hash is equally likely. See https://bitcoin.stackexchange.com/a/939/60443

There really hasn't been more work done just because the value is greater, as long as it meets the current difficulty. Nickel and diming the work done by comparing block hashes would create many more forks, thus requiring more confirmations to be sure a transaction is final.