What would be the consequence of a constant network difficulty?
You would eventually be up against network propagation speed, since miners would get powerful enough to mine faster that the blocks can propagate across the network. The nodes with the lowest latency between each other would always be ahead with the longest chain, penalizing those with higher latency.

You're correct that the 21 million will be reached much sooner.

You can observe this on the testnet when the block times are high enough, after it resets to the initial difficulty and blocks come flying in multiple times a second (basically as fast as the network can handle it). For more info see Testnet difficulty change