rtp: Fix integer underflow that could allow remote code execution.

Fixes MSVR-11-0088
Credit:  Jeong Wook Oh of Microsoft and Microsoft Vulnerability Research (MSVR)
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
