webmdashenc: Add better error handling

Return appropriate error codes and propagate the error codes from
helper functions to the outer calls. Also fix a potential leak in
call to av_realloc.

Signed-off-by: Vignesh Venkatasubramanian <vigneshv@google.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
