utvideoenc: fix theoretical integer overflow

The image sizes needed for this overflow to happen are currently not
supported in lavc, thus this should have no effect execpt making the
code more robust in light of future changes.

Fixes CID732245
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
