latmenc: error out when packet size is too large.

Previously it would just silently write out incorrect data.
This also fixes a potential integer overflow in the allocation.

Signed-off-by: Reimar Döffinger <Reimar.Doeffinger@gmx.de>
