Weird artifacts in my ray tracer
When spawning the refraction/transmission ray, don't adjust the ray origin by adding the normal: adjust it by subtracting the normal. This moves the ray origin inside the object, so it won't immediately self-intersect. Your current problem is caused by all your refraction rays immediately hitting the surface.

Don't forget that for accurate refractions, the ray needs to be able to intersect with the inside or back face of the object as well. If you're intersecting back-facing surfaces, you need to flip the normal before shading or adding/subtracting it to the origin.