Which algorithm should be used to replace SHA512withRSA? We are working on the web application that is basically generating a self-signed certificate during the installation. It's a Java based app and for generation of the certificate we are using SHA512withRSA. Can I ask what would be the best replacement for it? Something with better encryption level and more 'modern' security?

Any help would be appreciated.

--
Best regards
macosxgeek
In standard Java libraries, SHA512withRSA designates signature per RSASSA-PKCS1-v1_5 with SHA-512. There is no known attack when used with a sufficient RSA modulus size, and SHA-512 is very secure. Given that SHA-512 has a 19-byte DER DigestInfo prefix, and that PKCS1-v1_5 signature further requires 10 bytes and 1 bit more for the modulus, we can tell from the question that the RSA public modulus size is at least 1+(10+19)×8+512=745-bit, but that's far from satisfactory.

The only truly indispensable thing to do is check that RSA public modulus size. Anything below 1024-bit is unsafe, anything below 2048-bit is to upgrade ASAP, when performance is no issue 4096-bit is reasonable, unless we care about quantum computers usable for cryptanalysis, or fairies (there's abundant literature about both).

It might be useful to also switch to RSASSA-PSS (which has a better security argument), but that's a bigger change (including in Known Answer Tests, since RSASSA-PSS signatures change randomly for the same data). It should also be ascertained that MFG1 in RSASSA-PSS does not use SHA-1; that would not be insecure, but is criticizable nevertheless, and often creeps in upgrades from RSAES-PKCS1-v1_5 to RSAES-OAEP in Java code, see this.