How can I build a versatile camera rig which can both orbit around its focus and pan?
You need to be able to orbit at different distances and that means you need dynamic parenting. The way to achieve this is through Child Of Constraints. To animate this, you need to do the following in this specific order:


animate the camera into the start position of orbiting, till now the camera has no parent, it is free
place Child Of Constraint onto the camera, target is the orbit control empty, press Set Inverse button - now the camera is parented to empty
animate the orbiting motion by rotating the empty and any local motion of the camera (panning while orbiting etc..)
on the last keyframe of orbiting, insert a keyframe for the camera (LocRot) and a keyframe for the influence of the Child Of Constraint (100%).
move 1 frame forward
select camera and choose Object > Apply > Visual Transform, do not move the timeline
keyframe the Child Of Constraint to 0%
keyframe the camera (LocRot)
you can move the timeline now
the camera is now free again, without any parent
repeat and profit, camera will have as many parent constraints as there are orbiting motions




This is a bit tedious, if you are up for the rigging job you can put this into a script that can be run with a custom button in UI to dynamically parent and un-parent.

There is this addon which for simple use cases will work fine, but it is limited. This is very tricky to setup inside the hierarchy of an armature (for example dynamically parenting hand to thigh) when all is also parented to the Root (the trick is to avoid double transforms with some clever drivers).

And if you are trying to do this on a linked armature, it is yet again even bigger challenge, because after file reload all drivers on constraints that are not present in the original rig, but are present in the animation where the rig is linked are discarded.

For completeness and best result I am also plugging this in:

How to smooth camera animation to have fluid feel?