Paint peeled on fascia board, how to get paint to stick?
Paint will not stick to silicone.
You will need to sand it all off, be sure you do not leave any silicon on the board.
If you need to do more patching use an exterior Vinyl patching compound
Patch, sand , prime   and paint.