Why do the A320’s primary flight controls have hydromechanical backups only for the rudder?
The mechanical backup is only meant to maintain basic controllability during a temporary failure of the whole fly-by-wire system. As the OP mentioned, the rudder is deemed sufficient for basic roll control, while the horizontal stabilizer is deemed sufficient for basic pitch control even without any stability augmentation. That being said, the mechanical backup is not meant to achieve continued safe flight and landing (I would be really surprised if the mechanical backup meets all of the ARAC minimum controllability guidance); it's expected that at least one flight control computer can be rebooted to take over control after the initial failure (Ref. Airbus A320 Crew Manual). 

The reason for relegating the mechanical backup to rudder and trimmable stabilizer, instead of to elevator and ailerons, pretty much has to do with the choice of the passive sidestick as the pitch and roll inceptor. It lacks mechanical linkages to translate its motion to external assemblies, nor does it have a feel unit to provide feedback for the airloads. This leads to significant weight savings and simplicity in design, and is the primary advantage of the passive sidestick.

I suspect that the reason for even having a mechanical backup scheme is due to mistrust in FBW systems for civil aviation when they first came to use. As trust in redundancy and dissimilarity grew, newer generations of Airbus aircraft (e.g. A380, A350, A220) did away with it completely, and instead have an electrical backup servo control loop on each of the axis.