How much can bike shift cables spool, and over what radius?
How do you design a shifter and stay on the right side of this?

You don't.
You design a shifter that with repeated use breaks the cable. Then you expect the user to change the cable whenever it breaks, limping home without gear shifting function if the cable fails during a ride.
David Gordon Wilson has written a text about cable fatigue. According to it, the diameter ratio should be 72 for long life and 42 at a minimum. With 1.2 mm gear cable, it means 50.4 mm diameter or 25.2 mm radius at a minimum. Bicycle shifters have a far tighter cable bending radius.
The reason this works is because shifting is not safety critical in the manner brakes are. Bicycle brakes too have small bending radii in their cable path, but not as tight as shifters. In brakes, some design effort is needed to not immediately fatigue the cable. The brake cables still fail despite this.