Why does the pH of soft drinks get lower as they go flat?
I think you said it... phosphoric acid. Carbonic acid is "buffering" phosphoric acid... test seltzer water (composed of water and carbon dioxide with no other additives).

Another way of thinking about this is to consider carbonic acid (nominally a weak acid) to actually be a base. Looking at the pKa (the pH at which half of the acid is dissociated) is the best way to describe this.

The pKa of Carbonic acid is 6.35 

The pKa Phosphoric acid is 2.16

pH is basically a measure of free (dissociated) H+ in water... so as CO2 is removed, it is not available to act like a buffer and bind with H+... so the free H+ increases.