What is the optimal screw drive hole shape to get the maximum torque for a steel screw?
I would say the optimal screw hole shape that will allow the maximum amount of torque to be applied, would be no hole. 

Removing material from the head of the screw; no matter what shape, will weaken the screw slightly. The only way to maintain the full strength of the screw; and therefore achieve maximum torque, is to not remove any material. So a hex head screw (not to be confused with a hex drive screw), would allow for maximum torque.