Why does the retiree to contributor ratio matter when discussing the solvency of Social Security?
Pay as you go vs Fully funded pension plan distinction is relevant here. Also known as Defined benefit vs Defined contribution. 

If everyone saved for their own pension via payroll taxes or otherwise (Pay as you go/ Defined contribution), then the ratio of retiring to contributing people indeed would not matter. 

However, most governments pay current retirees from taxes on current workers. If there are fewer current workers relative to retirees, then either each has to pay more tax (discouraging working, eventually leading to declining tax revenue) or each retiree gets less pension. Governments typically do not save the current taxes to pay pensions when the current workers retire. Instead, governments are typically borrowing against future tax revenue. 

See also the Overlapping Generations Model (OLG).