wmaenc: check final frame size against output packet size

Currently we have an assert() that prevents the frame from being too large,
but it is more user-friendly to give an error message instead of aborting on
assert(). This condition is quite unlikely due to the minimum bit rate check
in encode_init(), but it is still worth having.
