What is the downside of not using Proof of Work for validator selection?
Proof of Work is needed to inventivize miners to not fork and, if there is a fork, some metric for determine which fork is the real blockchain (assuming no consensus changes in the fork).

Proof of Work incentivizes miners to not fork (i.e. see someone else's block, ignore it, and create their own block at that height) because of cost. It costs resources in order to produce a fork and stick with it. If their fork has no value, then they are wasting money on something that is worthless. However with other consensus systems like Proof of Stake, there is no cost to make a fork. Making a fork does not cost anything as there are no resources for you to spend to make the fork. In some systems, you can make a fork and still be mining/staking/minting/whatever on the original chain at no extra cost and no opportunity cost.

Proof of Work also provides a metric to determine which chain is the correct chain. Given two blockchains with the same consensus rules, the one with the most work (the sum of the difficulties of all of its blocks) is the one that is considered "the blockchain". In other consensus systems like Proof of Stake, no such metric exists for determining "the blockchain". This results in centralization as someone or something is needed to say "this blockchain is the right one". In some coins, a signer (usually the developer) must sign all blocks for the block to be considered valid, so that signer determines what "the blockchain" is and is a point of centralization.