avformat/rtsp: check memory allocation

Reviewed-by: Reimar Döffinger <Reimar.Doeffinger@gmx.de>
Commit message by: Benoit Fouet <benoit.fouet@free.fr>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
