avcodec/pixlet: Simplify nbits computation

Fixes multiple integer overflows
Fixes: runtime error: signed integer overflow: 1 + 2147483647 cannot be represented in type 'int'

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Reviewed-by: Paul B Mahol <onemda@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
