target-i386: Rearrange processing of 0F AE

Rather than nesting tests of OP, MOD, and RM, decode them all at once
with a switch.  Also, add some missing #UD checks for e.g. incorrect
LOCK prefix.

Signed-off-by: Richard Henderson <rth@twiddle.net>
