qdm2: check and reset dithering index per channel

Checking per subband would have the index exceed the
dithering noise table size.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
