dnxhdenc: fixed signed multiplication overflow

The low 32 bits of a multiplication are the same for signed
and unsigned operands.  Casting to unsigned before multiplying
is thus equivalent while avoiding signed overflow, which is
undefined by the C99 standard.

Signed-off-by: Mans Rullgard <mans@mansr.com>
