dvbsubdec: improve error checking

Use av_mallocz_array instead of iterating and check the returned memory.
Check returned memory and cleanly exit in case of error during the loop.
Avoid a null pointer dereference for invalid data.

CC: libav-stable@libav.org
Bug-Id: CID 29575
