tiff: Check the check_size() return value and forward it

Also use the same type for add_entry and check_size.

Bug-Id: CID 700699
CC: libav-stable@libav.org

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Signed-off-by: Vittorio Giovara <vittorio.giovarao@gmail.com>
