vorbisdec: avoid invalid memory access

This fixes some invalid memory access caused later in the function
by res_chan[] not being set for all channels. This happens when a
channel doesn't appear a submap. This change simply returns a
decoder error when this situation is detected.

Signed-off-by: Ronald S. Bultje <rsbultje@gmail.com>
