Why did Bitcoin Core switch from wallet.dat, cold.dat, lukewarm.dat, etc., into "sub-dirs"?
The original move to a directory per wallet file was motivated due to an issue with the way Berkeley DB behaved when multiple databases were opened in the same database environment, which are tied to directories containing the database. In particular, the issue is that BDB creates a database environment log which contains data from all of the databases open. This creates issues with backing up and portability as database logs need to be backed up as well.
With the transition away from BDB to SQLite, it is possible that we move back to the old way. However current discussions around that seem to indicate that that move is unlikely.l