Can a bad actor omit an unspent address from a pruned chain? This isn't a problem today, because if I want to run a pruned node then I still have to download the whole chain, but it sure would be nice if I could just download a pruned chain instead and validate it. 

The attack I'm thinking of would be that I download a pruned chain, but the person I got it from omitted an unspent address from a thousand blocks back. I'd have no way of knowing. 

This means that the full chain will always have to be downloaded and pruned manually. Right?

edit: I think I figured it out. All the unspent addresses on a pruned chain should add up to all the bitcoin in circulation, which is known based on how many blocks there are. So you could tell right away if someone evil omitted an unspent address.. Turns out that doesn't help
This is precisely why Bitcoin Core does not support pruning in this manner. However this does not always need to be the case. There are people working on things like UTXO set syncing where each block commits to the UTXO set as of that block. You can then use that to be sure that you have downloaded the proper blockchain.