Plain text size limits for AES-GCM mode just 64GB? Based on NIST SP 800-38D section 5.2.1.1, it seems that the maximum length of plaintext is 2^39-256 bits ~ 64 GB. 



We've got 100+GB files in genomics that need to be GCM encrypted so are concerned about hitting this.

So two questions:


What's the source of this rather low limitation?  
What happens (mathematically, not implementation wise) if you do cross this limit?
The source of the limitation lies in the fact that GCM has a fixed block counter using a 32-bit integer. Since the block size is $2^7$ bits, the total amount that can be encrypted with the CTR component is $2^{39}$ bits.

The first limit reducing this by 128-bits is the fact that the block counter starts at 1 and not 0, at least with a 96-bit nonce. Nonce sizes other than 96-bits are know to have reduced security. The second limit reducing this by 128-bits is because the CTR component is also used to encrypt the final GHASH prior to tag output.

Implementation wise, you should not be allowed to cross this limit. In theory, the security breaks down. There are different security limits for the amount of data encrypted, and for the amount of tags generated. With a long message, an $n$-bit tag provides $n-k$ bits of authentication security for a $2^k$ block message, in your case this brings it down to around $2^{63}$ with a file between 100 and 128 GB when used with a 96-bit tag, much lower than the $2^{128}$ one may expect. The worst case scenario is recovery of the hash subkey $H$, allowing forgery of tags with the same key.

Encrypting a full size 128-bit tag with a separate key in ECB mode should eliminate that problem, as the bits will no longer be malleable. That can be done as a wrapper around a GCM implementation with minimal modification and almost 0 overhead. The implementation would need to be modified to use a larger block counter, and the nonces would need to be shorter. You would no longer be able to call it GCM, but would be able to make use of all the features and code that make GCM implementations fast and timing attack resistant. A 32-bit nonce and a 40-bit block counter (with no additional GHASH) would allow almost 16TB per nonce, without the restriction of changing keys for every message if you encrypt the tag with a second key. If you even consider something like this, a professional consult on both the modifications and the code to do it would be highly recommended, new security proofs would need to be developed.

Other solutions to the problem including not using GCM, but a different mode like Poly1305-AES. OCB mode also has a suggested limit of 64GB per key, CWC mode has the same limit as GCM.