Relative Solubilities of Salts in Acidic Solution
Chloride, Bromide, and Sulfate all correspond to strong acids, while phosphate corresponds to a weak acid.  

With 1M H+ available, phosphate will be essentially fully protonated, while the others will still be anions.