using orientation sensor data to predict image points
It turns out that I had two bugs that broke my experimental results:


Even though the rotation matrix reports the rotation from Earth to Device, I should have been applying the opposite rotation to the point. It took me a day to convince myself of this.  Basically, if you rotate the whole frame about, e.g., $+x$, then coordinates that remained fixed w.r.t. the device will rotate w.r.t. the Device frame about $-x$.
I had a wrong minus sign in the rotation about the y-axis in the experimental results that I reported at the end.  Note that y-axis rotations have a different +/- pattern than x- and z- axis rotations.  See wikipedia for basic rotations.


The method described herein works fine (at least in theory).  I'll post an update after I port my python dummy code into Android and see how it works.