Transforming a ray from camera space to world space I am writing a Raytracer and generating rays from the camera. I have a working program but slightly confused about the linear algebra concepts involved in transforming the ray from camera space to world space. I already have the camera position in world coordinates. So, only need to transform the direction vector. I think there are two ways to achieve this,
a) Directly transforming the direction vector:
The direction vector can be built as,
direction vector = point on film - camera position
The camera is at the origin, so the above would reduce to the coordinates of the point on the film. Now, we can write this direction vector in homogeneous form and apply the camera to world transformation. This should give us the direction vector for the ray in world space.
b) Transform the origin and point on film:
The origin's position vector is at zero and the position vector of a point on the film is the point's coordinates. We can apply the camera to world transformation to these and this would give us two position vectors in world space: the camera's position vector and the film point's position vector. Then, the direction vector in world space can be directly built as,
direction vector = point on film (world space) - camera position (world space)
Mathematically, the first approach is to multiply the direction vector in camera space by the camera to the world (4x4) matrix, and this would be my answer. However, based on the second approach, I should do the same thing (multiply by 4x4 matrix) and then subtract by the camera's position in world space.
I tried programming both strategies and the second approach seems to give the right answer. What's wrong with my first approach?
Now, we can write this direction vector in homogeneous form and apply the camera to world transformation. This should give us the direction vector for the ray in world space.

That is the problem. When transforming a direction, you must ensure the w coordinate is 0, so that it doesn't get translated by your camera-to-world matrix. What you're doing in the second method is that you're transforming the direction vector as a point, and then subtracting the camera's position, which effectively undoes the translation.