FFmpeg produces high system load when downscaling 1920x1080 to 320x240
As @Mulvya said, nearest neighbor is even faster, but will produce bad quality. Nothing resampled, so you can't get faster operation.

But even nearest neighbor and can cause 100% CPU load, it's depends on how much you pump information through it. If you put very large image or 1000fps, you always will get high CPU usage.

Only one thing you may be interested: colorspace. Try to experiment with different color spaces, like yuv420 or rgb32. Maybe some versions of scaler can be faster.