What prevent a smartphone wallet from downloading my keys
Nothing.

A Bitcoin wallet that is online and connected in some way to the Internet is referred to as hot wallet.

That is why it is recommended to keep as much money as you would put in your pocket in a hot wallet.