Invalid challenge-ciphertext in IND-CPA and IND-CCA games Consider the IND-CPA game between a challenger $C$ and an adversary $A$ for  a given public-key encryption scheme $PKE$:


$C$ generates a key pair $(pk,sk)$ based on some security parameter $n$, and publishes $pk$ to $A$. $C$ retains $sk$.
$A$ may perform a polynomially bounded number of encryptions or other operations.
Eventually, $A$ submits two distinct chosen plaintexts $m_0,m_1$ to the challenger $C$.
$C$ selects a bit $b \in \{0, 1\}$ uniformly at random, and sends the challenge-ciphertext $c=E(pk,m_b)$ (the encryption of $m_b$ using the public-key $pk$) back to $A$.
$A$ is free to perform any number of additional computations or encryptions. Finally, it outputs a guess $b'$ for the value of $b$.


The advantage of $A$ is $Pr(b=b')-1/2$.

The IND-CCA2 game is the same except that, in phases 2 and 5, $A$ has access to a decryption oracle, with the condition that $A$ can not ask for the decryption of the challenge-ciphertext.

My question is: Suppose that there is an adversary $A$ that has a non-negligible advantage in one of these games. Suppose that, in phase 4, the challenger $C$ sends an invalid ciphertext to $A$ (i.e., $C$ sends a ciphertext that is not the encryption of $m_0,m_1$ or any other message $m$). In this situation, what does $A$ answer? Does $A$ knows that the challenge ciphertext is invalid?

Thanks in advance!
$A$ outputs a bit depending on whether it thinks you gave it an encryption of $m_0$ or $m_1$.

If you cheat and give it something completely different it has no usable information but still needs to output a bit.

If your PKE scheme has invalid ciphertexts $A$ might be able to detect that or not depending on the specifics.

So if you want to exchange your encryption of $m_0$ or $m_1$ for something else in a proof you have to show that $A$ cannot distinguish between these two situations.

Intuitively you can think about it this way:
If $A$ had some way to distinguish between the real security game and your simulation it means that your simulation looks different. So how do you know that whatever $A$ did to attack the original game will work on your simulation if it doesn't look the same to $A$?