Multiple Servo, Move when not powered on
In theory the rest of the servos are not ON because the CD4511 decoder
  is not turning their MOSFETs on and I also have not turned the high
  side MOSFET ON for them either... the problem is that they DO move


I suspect this is caused by caused by power backfeeding through the 'unpowered' servos. Unlike LEDs which block reverse current, RC servos have a control IC (and perhaps a discrete H-bridge) which look like a forward-biased diode when connected with reverse polarity. 

Consider the simplified circuit below with just 2 segments and 2 digits. The digit and segment FETs are modeled as switches, and servos as lamps with parasitic reverse diodes:-







simulate this circuit &ndash; Schematic created using CircuitLab

Digit 1 and Segment A are switched on, supplying power to Servo1A. But there is another path - shown by the arrows - through Servo1B, Servo2B (reversed), and Servo2A. Servo2B will drop about ~0.6V, leaving ~5.4V to be shared between Servo1B and Servo2A. Either or both of these servos may move if their share of the voltage is sufficient to operate them.       

To prevent backfeeds you could put a diode in series with the positive lead of each servo. However it might be better to switch the servos' signal inputs rather than their power supplies. Multiplexing the servo signals would also prevent possible problems caused by signals being fed into unpowered servos with floating grounds.