rsd: limit number of channels

Negative values don't make sense and too large values can cause
overflows. For AV_CODEC_ID_ADPCM_THP this leads to a too small extradata
buffer being allocated, causing out-of-bounds writes.

Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
