Is this a better GCM? Using the keystream from the block cipher for both parts of the GMAC key (initialization and encrypting the auth tag) seems to produce a better mode of operation than GCM, as successful forgeries and nonce reuse only compromise the particular (key, nonce) pair, rather than compromising integrity for every message encrypted with the given key.

It seems surprising that I can improve upon a well-known and widely-used algorithm as GCM, so I would like to know if there are flaws in this approach.
Is this a better GCM?  Not really.

You can derive an independent GHASH key for each message instead of using the same GHASH key for many messages as AES-GCM does.  It won't reduce security.  It's costlier without hardware support: you can't reuse the same GHASH lookup table to quickly leak secrets through timing side channelsauthenticate multiple messages.

But it's not much of an improvement in security: the difference is relevant only in the case that an adversary has already succeeded in forging a message and confirmed by the receiver that the forgery was successful (or the sender has reused a nonce, which works out to be just like the forger had made a forgery with the same nonce).


In this case, with AES-GCM, the adversary can forge arbitrarily many additional messages by recovering the GHASH key.
In contrast, with crypto_secretbox_xsalsa20poly1305, if the receiver demands sequential nonces or otherwise rejects repeated nonces, the forger can't use this to forge more than one message because the receiver will expect a different key for subsequent nonces.


It makes a difference only if the cost of a single forgery is small and you're really worried about how many forgeries you accept, which is not relevant to most applications—usually we consider a single forgery to be fatal, and strive to ensure it never happens.

This method is essentially what NaCl does in crypto_secretbox_xsalsa20poly1305: choose the Poly1305 key as the first 256 bits of the pad generated by the stream cipher XSalsa20, as suggested by Tanja Lange in 2006.  Exactly the issue you describe is addressed in the NaCl validation paper on p. 30, where Dan Bernstein points out the real security comes from bounding the forgery probability below $2^{-100}$.  This method is also by TLS with the TLS_*_WITH_CHACHA20_POLY1305_* cipher suites defined in RFC 7539 and RFC 7905, updated in RFC 8439 and included in TLS 1.3 in RFC 8446.  Of course, Salsa20, ChaCha, and Poly1305 also avoid leaking secrets by avoiding designs that present conflicts between speed and security in software implementations.  More on the background and shortcomings of AES-GCM.