vqf: Make sure the bitrate is in the valid range

Even if the sample rate is valid, an invalid bitrate could
pass the mode combination test below.

CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
