I'm trying to localize points I see in a camera image in terms of azimuth and elevation and match points between shots.Individual shots should differ only in rotation around the camera's center (roughly).It's not working (as I'll explain below), and I'm looking for things I might be doing wrong.I've tried different things, but this is what I'm doing right now:Let's say I have tilted my camera (around the $x$-axis) by some angle $\alpha$ and rotated it (around the $y$-axis) by $\beta$.Assuming my lens gives me a perspective projection with a focal length of $d$, I can represent pixel $(i,j)$ as a vector$$ \vec{v}=d\cdot\vec{z}' + i\cdot\vec{x}' + j\cdot\vec{y}', $$where $\vec{x}', \vec{y}', \vec{z}'$ are the unit vectors of the standard basis rotated by $\alpha$ and $\beta$ around x-axis and y-axis, resp.$\vec{v}$ then represents the point in 3D space at which the photoreceptor cell in the CMOS was when it perceived the point in real space.Say $\vec{v}=(v_1,v_2,v_3)$.Then I should be able to calculate azimuth $\phi$ and elevation $\psi$ wrt. an unrotated take as:$$\phi = \mathrm{arctan}(\frac{v_1}{v_3})\quad\text{and}\quad\psi = \mathrm{arcsin}(\frac{v_2}{|\vec{v}|}).$$This doesn't work, though.That is, without tilting the camera it more or less does.But say I have a shot where both $\alpha$ and $\beta$ are not zero.Then, the points which appear in that shot and in others for which $\alpha$ or $\beta$ are zero don't come out with the same azimuth and elevation.It seems that they generally cluster too much around the center of that shot.One of the problems is that there are a lot of things that could get in my way here, like any combination of:a distorted camera imagewrong focal lengthwrong rotation anglesrotation around a point too far from the focal point.I'm looking for more potential sources of error and (in)validation for the approach outlined above.Here is a visualization of the outcome of the procedure so far: the white horizontal and vertical axis are azimuth and elevation, resp.The distorted rectangles are boundaries of indiviual shots.It's hard to explain what exactly the points are, but in essence points with the same color are identical in the real world and should therefore appear at the same absolute azimuth and elevation.These points form a distorted shape, not a rectangle.You can see that at each level of elevation, the points from each camera image appear more and more contracted around the camera image's center.
Pixel-To-Angle Transformation in Camera Image
linear algebra;computer vision;geometry