qdm2: fix signed multiplication overflow

This code relies on the result wrapping as for unsigned
values, and the sign is not used.  Thus an unsigned type
is proper here.

Signed-off-by: Mans Rullgard <mans@mansr.com>
