How do people know decryption is successful? Decryption is simply a process of transforming one byte string into another with an input argument private_key. However, if people have no knowledge about the plaintext, how do they know the decryption is successful?

For example, suppose the plaintext is a byte string abcd, and transformed into ciphertext whatever in the encryption process. Then let's assume the decryption process transforms whatever into abcd with one private key, and wxyz with another private key, how do people know which result is the original plaintext?
It depends on the encryption scheme if the correctness of the result is verified or not. 

For instance, the output of an asymmetric scheme such as raw RSA decryption (just modular exponentiation) is just a number; you cannot directly distinguish between a correct and incorrect answer.

However a secure padding scheme such as OAEP then a possible output of OAEP can be "decoding error". Decoding error shows that something went wrong when removing the padding, which would happen if the resulting (padded) plaintext message contains is not encoded correctly. And that situation would arise (with high certainty) if either the ciphertext or the private key was incorrect.



The same goes for symmetric decryption. Decryption with AES-CTR always succeeds, the result is just a bit string. AES-GCM is however an authenticated cipher and will result in an error if the key is incorrect.



There are schemes in which the result does sometimes result in an error and sometimes it doesn't. A perfect example of this is AES in CBC mode with PKCS#7 padding and PKCS#1 v1.5 padding for RSA encryption. In that case the scheme sometimes results in an error and sometimes it doesn't. This leaks information about the plaintext which can be exploited by a padding oracle attack.



If your scheme doesn't return an error (with high certainty) then you can protect it with a signature (for asymmetric crypto) or MAC (for symmetric crypto). Usually sign-then-encrypt is used for asymmetric crypto and encrypt-then-MAC for symmetric crypto.



As an attacker you can use any information about the (padded) plaintext to see if the result is (partially) correct. This could include information about padding, formats of data structures (and therefore files), frequency analyzes of text etc. etc. etc.

An attacker may also use errors returned from the decrypting party to obtain information. This is called a plaintext oracle attack (a padding oracle attack is usually seen as a specific plaintext oracle attack).



Notes:


an error is often indicated as $\bot$ in scientific papers on cryptography (falsum)