Percent composition by volume of the reaction H2 + Br2  2HBr
Your equation and your math appear correct to me: I also get $x=0.808$ from the quadratic equation, and calculate the same percentages.

As long as you are supposed to assume ideal gas behavior in the question, it appears to me that the answer key is incorrect.