Is there something wrong with a column camera matrix? I am writing a simple ray tracer. I was working on my camera matrix when I encountered a problem. When I changed the look vector, so that it headed down, the image, which was supposed to go up, actually went down as well. Likewise, if I changed the look vector so that it headed up, the image went with it.
I understand that the camera matrix's job is to change world coordinate into eye coordinate. Since I am using column matrices, here's my camera matrix, with $r$ being right vector, $l$ being lookAt vector and $u$ being up vector

$$
\begin{bmatrix}
r_x &amp; u_x &amp; -l_x &amp; 0 \\
r_y &amp; u_y &amp; -l_y &amp; 0 \\
r_z &amp; u_z &amp; -l_z &amp; 0 \\
0 &amp; 0 &amp; 0 &amp; 1
\end{bmatrix}
$$

This is the transpose of the gluLookAt matrix of openGL. I dont understand why openGL use their matrix though, since they are also using column matrices.
What you have there is the matrix that goes from eye space (camera space) to world space. You can see this by observing that, for instance, if you apply this matrix to the column vector $[1, 0, 0, 0]^T$, which is the right-vector in eye space, then it will return $[r_x, r_y, r_z, 0]^T$, the right-vector in world space.

You want the matrix that transforms the other direction, which would be the inverse of this one. And assuming $r, u, l$ form an orthonormal basis, the inverse will simply be the transpose.