Trying to extend the animation system
I got it working. Leaving the answer here in case somebody else is looking for a way to develop a similar kind of custom "armature animation modifier" (if I can call it that). Again if anyone sees a better way, let me know!

Basically, I dug a little deeper into why the armature_update_post Python callback (which I added to BKE_pose_where_is, as explained above) wasn't working reliably. The problem was that BKE_pose_where_is can be invoked from an arbitrary thread, and therefore the Python callback runs in an arbitrary thread. In this Python callback, I was checking bpy.context.window.screen.is_animation_playing but this doesn't work because bpy.context.window returns None if not called from the main thread. (See ctx_wm_python_context_get in context.c.) So instead, I decided to add yet another Python callback so that the UI can tell the script when animation playback starts &amp; stops, and let the script keep track of the playback state itself.

Here's a summary of the approach I'm using now (based on Blender 2.78):


Add a new Python callback armature_update_post and invoke it from BKE_pose_where_is in armature.c.
Add a new Python callback animation_toggle and invoke it from ED_screen_animation_play in screen_ops.c.
In rna_pose.c, add a call to BKE_pose_where_is_bone at the end of rna_PoseChannel_matrix_set, so that when a PoseBone's matrix attribute is set from Python, the bone is fully updated right away (specifically the chan_mat member of bPoseChannel). This was necessary because rna_PoseChannel_matrix_set seems to leave the bPoseChannel in a partially updated state. If you don't do this, the whole approach doesn't work.
Handle the callbacks in the Python script. In armature_update_post, if animation is playing, modify the PoseBone matrices however you want, in parent-to-child order (basically just scan obj.pose.bones sequentially).


That's just the general approach at a high level. As you can see, I wasn't able to implement a pure Python solution -- I had to hack in C a little bit. The Python callback runs kind of slowly, so the playback rate in Blender slows down. I might move the logic into C later to speed it up, but for now, at least it's working.