Is this RSA signing and verification process secure? I need to verify that messages in a communications program are from valid senders. Below is a highlevel outline of how I think the process would work. The connections themselves are done over TLS. The process here is just using RSA signing/verification on top of that. The data itself is actually not that important so the extra AES encryption is somewhat redundant. Can someone confirm whether this is the general approach or set me straight if not?

Sending host

plaintext = "Hello World!"

#SIGN
digest = SHA256(plaintext)
sig = rsa_encrypt(sender_priv_key,digest)

#ENCRYPT DATA
iv = generate iv
key = generate random key
ciphertext = aes_encrypt(iv,key,plaintext)

#ENCRYPT KEY
encryptedKey = rsa_encrypt(recipient_pub_key,key)

#SEND
send(sig+iv+encryptedKey+ciphertext)


Receiving host

#RECEIVE
sig+iv+encryptedKey+ciphertext = recv()

#DECRYPT KEY
key = rsa_decrypt(recipient_priv_key,encryptedKey)

#DECRYPT DATA
plaintext = aes_decrypt(iv,key,ciphertext)

#VERIFY
sigDigest = rsa_decrypt(sender_pub_key,sig)
msgDigest = SHA256(plaintext)

if sigDigest == msgDigest
    #valid sender so long as we can trust where we got the public key
    #data integrity is good
Comments already pointed out, that encryption and signing are not the same and should not be exchanged deliberately. Of course in practice specifically for RSA, not PKE in general, and only in the textbook variant (no padding), encrypt/decrypt are bascally the same operations as sign/verify: For all of them, you just do modular exponentiations; and the defining properties of the keys (the product of the public and private exponents equals $1$ modulo $\phi(N)$) are analogously.

But considering your algorithm, there are flaws. First, your algorithm for signing is actually this:


Hash the message
"rsa_encrypt"the hash with a private key


First, if we interpret "rsa_encrypt" just as modular exponentiation, this is still just textbook-RSA without any proper padding , which is vulnerable to certain attacks and usually the padding scheme PSS is recommended. (also see wiki)

And then you have the implicit assumption, that your function call actually is the plain modular exponentiation: If you call "rsa_encrypt" from a cryptographic library, are you actually sure that it is just the modular exponentiation? It could e.g. already have soem kind of padding in there, which might be unsuitable for signatures. Or did you implement the function yourself? Because then of course there is the old saying "don't implement crypto yourself" (and the question doesn't sound like homework), and for good reasons any proper cryptographic library will offer separate methods for signatures and encryption. In the implementation details, there are quite a lot more differences when it comes to side-channels, performance improvements, etc.

I would recommend, try to get rid of your current primitives, and use a cryptographic library instead, which gives you proper signature algorithms. Other than that, you are e.g. generating the random key for the symmetric part of the hybrid encryption. You shouldn't have to do that manually, because it is another possible weakness (you have to ensure that you're using a proper CSPRNG). But most cryptographic libraries should also have that functionality.