rtpenc: Cast a rescaling parameter to int64_t

This avoids overflow if frame_size is over 2147, since both
frame_size and AV_TIME_BASE are plain integers.

Signed-off-by: Martin Storsjö <martin@martin.st>
