Bytes and bits conversion?
That's because memory is usually byte-addressed.  The address that you load from is an address of a byte in memory.  Each address refers a different byte in memory.  For a 32-bit address, there are $2^{32}$ different possibilities for the address, so $2^{32}$ different bytes that the address could refer to (and no overlap among these bytes).  $2^{32}$ is equal to 4GB.

If we had a hypothetical machine where memory was bit-addressible, then it would indeed be 4Gb, i.e., 4 gigabits instead of 4 gigabytes -- but no one builds machines that way today.