proresenc: Realloc if buffer is too small

The buffer allocation may be incorrect (e.g. with an alpha plane),
and currently causes the buffer to be set to NULL by init_put_bits,
causing a crash later on.

So, detect that situation, and if detected, reallocate the buffer
and ask for a sample that shows the problem.

CC: libav-stable@libav.org

Signed-off-by: Diego Biurrun <diego@biurrun.de>
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
