libkvazaar: Replace asserts with proper errors

Changes function libkvazaar_encode to return proper error codes instead
of crashing when the video dimensions or pixel format change in the
middle of encoding.

Signed-off-by: Arttu Ylä-Outinen <arttu.yla-outinen@tut.fi>
