Should I calculate SHA-256 hash of a raw bytes of a token or it's hex encoding? I have a system where I should generate a secret token for a user. The presence of a token is sufficient to grant access to some user-related data.
I am generating a 4096 bit token from a cryptographically secure random number generator. Internally I'm handling the token as a string, encoded to hex. Token is hashed using SHA-256 and this is stored in the database, hex string representing the token is sent to the user and never stored.
When the user presents the hex string of the token, I calculate its SHA-256 hash and look it up.

Question is: am I sacrificing something by calculating SHA-256 of the hexadecimal encoded string vs calculating the hash of the raw byte payload? I'm writing this in Go, so I'm really calculating the SHA-256 hash of []byte(str) which converts the string to byte array.
I have this feeling I'm comparing apples to oranges to melons perhaps.
Should I perhaps get the hex encoded string from the user, convert it to byte array, hash that using SHA-256 and then use that?
Question 1-a: Do things change if it's base64url instead of hex?
The encoding doesn't change the output of the entropy source. They are reversible operations. You can use what encoding suits you, change to base64 for transmission and strong on the database, and change to byte is a good choice since, in general, the cryptographic hash functions are accepting bytes to process, so it might better to convert them bytes before applying SHA-256 so that you can get the benefit of the speed of the SHA-256. Base64 or Hex may need more time to hash the same data due to the increase in the length, that is $~3n/4$ in Base64, and $2n$ in Hex for $n$ bytes.
If the CSPRNG has good entropy then you even don't need the SHA-256, or at least 4096-bit token, hashing the output of the source is a good idea, however, if the source needs 4096-bit to produce good 256-bit randomness, there might be a problem there. This, however, really depends on your CSPRNG.