Calculating equilibrium concentrations of silver and halide ions
Both chloride and bromide ions are present in 10-fold excess over silver ions. That means that the chloride and bromide concentration in solution will not drop by much (they will remain major species).

The solubility product of AgCl is 200-times higher than that of AgBr. If both AgCl and AgBr precipitate, the chloride solution would be 200-times higher than that of bromide. That is impossible because so little silver ions are available, and bromide and chloride start at equal concentrations. Thus, no silver chloride will precipitate (if this explanation is not convincing, suspend your disbelief until we can check the equilibrium concentrations).

Most of the silver ions will precipitate. As a first approximation, let's say the silver ion drops from 5 mM to zero, so the bromide concentration will drop from 50 mM to 45 mM. Of course, this can't be the equilibrium, so we can dissolve some of the AgBr again.

$$[\ce{Ag+}] = \frac{K}{[\ce{Br-}]} \approx \frac{\pu{1e-11}}{\pu{45e-3}} = \pu{2.2e-10}$$

So the concentration of silver ions is roughly $\pu{2.2e-10 M}$, that of bromide ions roughly $\pu{45 mM}$ and that of chloride ions $\pu{50 mM}$, with no silver chloride precipitating.


  At equilibrium condition, both AgCl and AgBr would have precipitated out.


Now that we have solved the exercise, you can verify that AgCl will not precipitate. The reason is that the silver ion concentration is extremely small due to the presence of 45 mM bromide ions. In fact, if NaCl and AgNO3 had been mixed first, AgCl would have precipitated. However, adding NaBr would have dissolved it again eventually, leading to the same final state.


  However, this doesn't match the answer key. Where am I wrong?


As mentioned in the comments, the set of final concentrations does not make sense from a stoichiometry (charge balance) perspective, and also contradicts the claim that AgCl is saturated. If you assume that both AgBr and AgCl have formed and attempt to solve the problem by brute force, the solutions to the two equations with two unknowns will only yield solutions that are not attainable (negative concentrations, or concentrations higher possible with the available starting materials).