Don't return 0 if buffer setup failed. That signals the RTSP demuxer that
the packet was filled in, leading to virtually random behaviour in the
decoder later on. Instead, return a negative value.

Originally committed as revision 21851 to svn://svn.ffmpeg.org/ffmpeg/trunk
