Replace light switch connected to a power outlet with dimmer - two black wires to one black and red
Yes, go ahead and replace the receptacle with a GFCI receptacle

Your idea of replacing this receptacle with a GFCI is a good call, considering that Code has required GFCIs in bathrooms for a good few decades now.  You'll want to remove the current pigtail since your dimmer has wire leads, by the way.  Also, leave the sticker on the GFCI's LOAD terminals since you have no use for them here.

Once that's done, the dimmer is easy

The dimmer's black lead gets bundled in with the always-hot coming into the receptacle (you may need to pigtail the receptacle's hot for this), the black wire going out to the cable to the light gets connected to the red wire on the dimmer, and the dimmer's green wire gets bundled in with the grounds in the box.