emulating AES GCM with AES CTR Building off of Relationship between AES GCM and AES CTR... if your GCM nonce is "aaaa aaaa aaaa" (no spaces) then the CTR IV is that with (0,0,0,2) appending to it.

My question is...  what happens when the counter gets to (255, 255, 255, 255)? Would the last five bytes be (98, 0, 0, 0, 0) (where 98 = b) or would they be (97, 0, 0, 0, 0) (where 97 = a)?

I'm guessing the latter but idk.

Thanks!
In the NIST Special Publication 800-38D page #8 the maximum inputs lengths are defined in the standards as;


  The bit lengths of the input strings to the authenticated encryption function shall meet the
  following requirements: 



$len(P) \leq 2^{39}-256$


and in section 8.3 the number of invocations requirements for all implementations:


  The total number of invocations of the authenticated encryption function shall not exceed
  $2^{32}$, including all IV lengths and all instances of the authenticated encryption function with
  the given key. 



As a result, the recommendation doesn't advice the overflow of the counter.


We can find the maximum as:

$2^{32}$ invocation, per invocation you can encrypt 128-bit ($2^7$-bit) with AES, thus

$2^{32} \cdot 2^{7} = 2^{39}\text{-bit} \approx 68.7\text{GB}$ is the maximum data that one can encrypt with AES-GCM under the NIST recomendation.