h264: vdpau: fix crash with unsupported colorspace

The h264_vdpau decoder crashed if output colorspace was not 8-bit 420.
Add a check to error out instead (current hardware does not support
other colorspaces, so successful decoding is not possible).

Signed-off-by: Martin Storsjö <martin@martin.st>
