vp9: change type of tile_size from unsigned to int64_t

Otherwise the check 'tile_size < size' treats a negative size as
unsigned, causing the check to pass. This subsequently leads to
segmentation faults.

This was originally fixed as part of Libav commit 72ca83, so the
original author is one of the following developers:
        Anton Khirnov <anton@khirnov.net>
        Diego Biurrun <diego@biurrun.de>
        Luca Barbato <lu_zero@gentoo.org>
        Martin Storsjö <martin@martin.st>

Reviewed-by: Ronald S. Bultje <rsbultje@gmail.com>
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
