avcodec/mss3: Change types in rac_get_model_sym() to match the types they are initialized from

Fixes integer overflow
Fixes: 1372/clusterfuzz-testcase-minimized-5712192982745088

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/targets/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
