Will I have to install new cables when converting from a SD CCTV system to HD?
Coaxial cables are good for pretty much any resolution. So a full HD security camera will probably be fine over your existing coax. After all, your HD television signal probably arrives via coax to your TV or STB.

What you will need at the other end of the cable is a receiver that can record full-hd images at a reasonable FPS over the number of channels you have active. If you have HD cameras and a recorder that can record your HD images, then you should be good to go.