How to handle specular highlights with reflection / total internal reflection in a ray tracer? Let's say that I have a raytracer that is rendering a scene that includes a refractive object.

When a ray hits the refractive object, I know that I can use the fresnel equation (Schlick's approximation in my case) as well as the object's base reflectivity amount to figure out how much light is reflected or transmitted at that point on the surface.

At this point, I can get the reflected color using a reflection ray (possibly recursively), an environment map, and also calculate a specular highlight for the light source(s) I have.  I sum whatever reflection colors I have, multiplied by the amount of reflection from the fresnel equation.

Next, I calculate what amount of light comes in due to refraction by following the refracted ray (recursively) and multiplying that value by 1.0-reflectionAmount.

That makes sense to me so far, but where I get a little confused is for the refraction rays.

When I follow the refracted ray into the object and hit the back side of the surface the first time, i once again use the fresnel equations to calculate the reflected (internally) and refracted multipliers for the light, taking into account the possibility of total internal refraction (assuming here that the inside of the object has a higher IOR than outside of the object, an ok assumption for my case).

But, at this point, do i apply a specular highlight to the surface that i hit from the inside?

On one hand it's a surface so i feel like i should, but which ray should i use for the lighting - the refracted ray or the reflected ray, or both?

If using the refracted ray, it feels wrong to use the direction to the light source for the specular light, and feels like i should shoot the refracted ray to the surface of the object and see what the specular highlight would look like from there and use that - but do so recursively, using the fresnel equation again at each step.

Can anyone explain a decent way to handle a specular highlight in a ray tracer that supports refraction and total internal reflection?

Since this is a ray tracer, not a path tracer, instead of being limited to physically accurate solutions, I'm looking for any plausible solutions, but of course, physically inspired solutions are likely to be better!

Thanks!
But, at this point, do i apply a specular highlight to the surface that i hit from the inside?


Unless you have a light source inside your object, there's no point in doing lighting on the inside surface as you'll hit the object again before you reach the light or environment.

If you want to support a light in the object, then by all means do your lighting but make sure to trace rays to check that you actually reach the lights without exiting the object.

Otherwise, that lighting should come from your (internal) reflected ray which will eventually hit the surface again, be refracted and get light from the outside.


  which ray should i use for the lighting - the refracted ray or the reflected ray, or both?


I think the confusion here is that when you write of "calculate a specular highlight" there is an assumed BRDF which may not make much sense for refraction. At least probably not without a few tweaks. What you really need is a BTDF (bidirectional transmittance distribution function).

If you want to keep it simple, I think you could adapt a Blinn-Phong model for refraction by using the half vector of equation (16) in Microfacet Models for Refraction through Rough Surfaces.