What's wrong with RSA and OpenSSL?
To a cryptographer, "signing a document is to encrypt its hash using signer's private key" is wrong, because:


It is specific to RSA and cousin cryptosystems including Rabin, and not even remotely descriptive of other common signature schemes;
It misses an important step, padding, which is different for RSA signature and RSA encryption. If we use no padding, some attacks apply (in particular, that allow computing signature of a document from a moderate number of signatures of other chosen documents).


PHP's openssl_private_encrypt used in the software linked in the question is thus horribly named (it should be openssl_RSA_sign), but when used with OPENSSL_PKCS1_PADDING (which seems to be the default), I believe it uses RSASSA-PKCS1-V1_5, an RSA signature padding; it's a tad obsolete and lacking a strong security argument (contrary to RSASSA-PSS), but stands currently unbroken.

Note: the linked software uses SHA-1, and that's obsolete, and broken by the definition we use in cryptography. It was practically and publicly broken in 2017.

Note: the linked application is at best as worth of trust as the weakest of the web server running it, the web client used to access the website for signature and that used for signature verification, the cryptography between server and clients, the source of the public key, the generator of the private key and anything that was ever used to handle it.