mp3on4: ensure that the frame channel count does not exceed the codec channel
count.

This also allows for checking output data size based on the actual
number of channel instead of the maximum number of channels.
