flvdec: Check the return value of a malloc

The callers of this function can't report errors sanely. If this
one malloc fails, don't write the extradata byte, make sure we
try to malloc it the next time we're called instead, and make sure
we still consume the input data byte.

CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
