AES GCM : is it acceptable to return the wrong plaintext if the tag is incorrect? Let's start by saying I'm no cryptography expert, I'm just a developer, so feel free to correct me (using words, not downvotes) if what I'm saying is non-sense.
Context: I'm doing some crypto as a service for embedded devices. Users of this service don't have access to any key. They're only able to tell the device &quot;use the key &lt;index&gt; to encrypt/decrypt this&quot;.
According to the NIST (Chapter 7.2), when decrypting a cipher using AES GCM, if the tag is wrong, the cipher should not be returned :

if they are identical, then the plaintext is returned; otherwise, FAIL is returned

This raises two questions.
First: how bad is it to return the wrong plaintext anyway? (but still returning an error, it would be up to the user to check it). Is there any mathematical reason for that? Could an attacker somehow leak the key by trying to decrypt forged ciphers?
Second: if the answer to the first question happens to be &quot;it doesn't matter whether you return the wrong plain or not&quot; then this is not an issue anymore. But otherwise: our AES driver is written in C, how can we prevent the cipher from being returned in case the tag is wrong? As far as I understand, to compute the tag the decryption process must be done entirely. The thing is the destination buffer of the decrypted cipher is a shared memory a potential attacker could have access to. Having a temporary buffer to store the resulting plaintext in order to compute the tag is not considered as it would not fit the device memory.
There is an article* that answers the question in the negative for GCM and CCM. The article introduces the first formalization of the Releasing Unverified Plaintext (RUP) setting. The related security notion is the Ind-RUP.
The security question is can an adversary forge messages with unverified messages? In this game, confidentiality is not relevant, since in the game the adversary is accessing the plaintext of the chosen messages. AES-GCM fails to satisfies this.

How to Securely Release Unverified Plaintext in Authenticated Encryption by Elena Andreeva and Andrey Bogdanov and Atul Luykx and Bart Mennink and Nicky Mouha and Kan Yasuda


AE schemes such as GCM [34] and CCM [49] reduce to CTR mode in the RUP setting. This is because the adversary does not need to forge a ciphertext in order to obtain information about the corresponding (unverified) plaintext.


Your questions:

It is very dangerous. AES-GCM internally uses CTR mode and bit flipping is very easy in the CTR mode. An attacker can easily change the ciphertext bit on their advantage. This is catastrophic if the structure of the message is known.
One must never ignore the tag invalid!
It is not about the leak of the key, CTR mode is IND-CPA secure. It is about changing the messages. This is why CTR mode can't have IND-CCA. GCM provides integrity and authentication and authenticated encryption &gt; IND-CCAs

For the second part you may benefit from is Q/A which provides a solution for the limited chunks for the GCM;

How to securely encrypt/decrypt data with a maximum chunk size?




Also for general guide

What are the rules for using AES-GCM correctly?, and
Deep into bounds Does GCM (or GHASH) only provide 64-bit security against forgeries?


*Thanks to Squeamish Ossifrage for pointing the article.