Bitcoin Core LevelDB Index saves forked blocks
They're kept because it is complicated to remove them (it would require rewriting the block storage files), and there is hardly any cost to keeping them (they're a small fraction of the total). Furthermore, close to the tip we want to have forks available, in order to be able to reorganize quickly if one of them ends up becoming the main chain.

More technically, the $DATADIR/blocks directory for Bitcoin Core is not a representation of the blockchain. It's a store of all valid blocks we've ever downloaded. Which out of them end up being considered the main chain is a different issue, and dealt with independently.

Also note that in pruning mode all old blocks are removed - including these small forks.