vmnc: check that subrectangles fit into their containing rectangles

Fixes possible invalid writes with corrupted files.

CC: libav-stable@libav.org
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
