How to install a house number plaque on vinyl siding?
I would just use silicone, as you can you can just rub it off after it is done (I would expect a small bit of damage but nothing that destroys integrity).  You might need to tape it until the silicone is dry if the numbers are heavy.