H.264: Check the return value of decode_vui_parameters()

Files with invalid VUI are now rejected like
other invalid SPS are.

Fixes issue1231.

Originally committed as revision 19335 to svn://svn.ffmpeg.org/ffmpeg/trunk
