What's the actual difference between RSA encrypting and Descrypting and RSA Signing? this question arised because I'm working with Windows' CNG library and it has 4 functions for RSA:


BCryptEncrypt (private key, message)
BCryptDecrypt (public key, message)
BCryptSignHash (private key, hashed message)
BCryptVerifySignature (public key, signature, hashed message)


So I was wondering, is there any difference between "Encrypting + Decrypting" and "Signing + Verifying"?
I mean, if I hash the message and then encrypt it, wouldn't that be the same as Signing it? or Am I missing some logic that is behind that function? (talking about RSA in general, not about how windows' CNG work).

The only visible difference I see is:


For Encryption and Decryption I can use OAEP and PKCS1 padding.
For Signing and Verifying I can use PKCS1 and PSS padding.


But again, is this because of how those functions are made in this library? or because the "operation" and the output are done in different ways depending on if it is Signing or Encryption?

If they are different, then why do we need the Signing part? Isn't encrypting with private and decrypting with public secure enough to "sign" it?
So I was wondering, is there any difference between "Encrypting + Decrypting" and "Signing + Verifying"?  I mean, if I hash the message and then encrypt it, wouldn't that be the same as Signing it? 


First of all RSA or any public key is not preferred for encryption, we prefer a hybrid-cryptosystem where a key is transferred/exchanged with public-key cryptography and the message is encrypted with a symmetric encryption scheme with the transferred/exchanged key.

RSA encryption to be secure requires a padding scheme like PKCS#v1.5 or OAEP. The letter is preferable since there is a security proof. The modulus size prevents you from encrypting arbitrary sizes so you need to encrypt a message dividing part like in block ciphers then perform the slow encryption many times.

RSA signature with PSS requires hashing not only for the security but for the message size. There is also RSA Full Domain Hash (FDH) Signature that hashes and signs and it has existentially unforgeable under adaptive chosen-message attacks.

If you hash a message then encrypt, then except you, nobody can verify the signature. You need to use your private key for signatures even for RSA-FDH. Keep in mind that it is not advisable to use a key for both encryption and signatures.

So the main difference in encryption vs signature is the used key and padding requirements. If you encrypt a message without padding you will have many attacks like it is malleable and see more here Twenty Years of Attacks on the RSA Cryptosystem


  But again, is this because of how those functions are made in this library? or because the "operation" and the output are done in different ways depending on if it is Signing or Encryption?


Yes, the operations are done differently since they require different padding to be secure.


  If they are different, then why do we need the Signing part? 


Because they are different so one needs different functions. Even if they are the same one still needs these functions to remove the ambiguity. Think about there is only encrypt and decrypt functions and the developers need to tell everyone to use decryption for signature and encryption for verification. Some people will use the encryption for the signature that will reveal unwanted information. Unfortunately, this is due to the RSA that has both encryption and signature.


  Isn't encrypting with private and decrypting with public secure enough to "sign" it?


No, you need hash then sign for RSA-FDH and hash and apply PSS then sign.

signature = RSA-FDH(Hash(message))


of

signature = RSA-PSS(Hash(message))