Given the mass of precipitate, how can I calculate the relative amount of the original dissolved compounds?
x = number of moles of chloride
y = number of moles of bromide

Since you know the mass of the original $\ce{NaCl}$/$\ce{NaBr }$mixture and the molar masses of $\ce{NaCl}$ and $\ce{NaBr}$, you can write an equation of the form:

$58.44 \pu{\frac{g}{mol}}~ x + 102.89 \pu{\frac{g}{mol}}~ y = 0.180~ \pu{g}$

Based on this idea, you can write a similar equation for the final products $\ce{AgCl}$ and $\ce{AgBr}$. With that in hand you will then have a system of equations you can solve (for x and y) that can be converted back into the masses of those salts.