Is history reversal a concern if the absence of attackers is assumed?
Yes, there is still a need to wait for confirmations.
Even in the absence of active attackers, there are many reasons the Bitcoin network could diverge, such as:

Large scale internet routing issues - we've already seen these with BGP routes being incorrectly broadcast, leaving large chunks of the internet temporarily inaccessible from other large chunks. In such a scenario, if miners are isolated into one of the two groups, each isolated group will advance the network without any visibility into the blocks being produced by the other side until the two networks are merged
Implementation bugs - although Bitcoin nodes generally just run Bitcoin Core, different implementations of the same Bitcoin Protocol can occasionally result in consensus bugs that result in everyone on one implementation rejecting blocks from the others. This has resulted in a few forks on other chains with multiple common implementations, such as ethereum.
Just normal mining - the very nature of mining means that sooner or later, two miners will produce block N simultaneously. This results in a split chain until the next block is mined, at which point only the block that is included in the prevBlock for the new block is retained.