Is there something wrong with a column camera matrix?
What you have there is the matrix that goes from eye space (camera space) to world space. You can see this by observing that, for instance, if you apply this matrix to the column vector $[1, 0, 0, 0]^T$, which is the right-vector in eye space, then it will return $[r_x, r_y, r_z, 0]^T$, the right-vector in world space.

You want the matrix that transforms the other direction, which would be the inverse of this one. And assuming $r, u, l$ form an orthonormal basis, the inverse will simply be the transpose.