hevc_ps: check overflow and test alternate syntax

Some streams were found to have what appeared to be truncated SPS.
Their syntax seem to be valid at least until the end of the VUI, so
try that syntax if the parsing would overflow the SPS in the
conforming syntax.

Fixes ticket #3872.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
