flacdec: give a more accurate error message when validating channel
layout. differentiates between invalid values and unsupported values.

Originally committed as revision 18153 to svn://svn.ffmpeg.org/ffmpeg/trunk
