Outdoor 12VAC LED lites flashing
Since all are flashing, this is a power supply problem.  (You can call it a transformer if you like.) 

The "flashing, then shutting off" behavior is typical of a power supply which is overloaded.  The power supply starts up, and its self-protection circuits discover that it's overloaded, and shuts the power supply down in what's called a "crowbar".  The power supply then reboots and tries again.  This results in the flashing.  It sounds like your power supply has an additional feature that it stops retrying after n number of tries.  

So your first solution is to reduce your load.  You already do this when only 3 lights are connected.  You can also put an ammeter on the line and see how much power is actually being drawn, and compare that to the spec on the power supply.  

Noting that your switch arrangement is highly unusual, I have to wonder if something about it is causing more amps to flow than intended. I.e. A miswiring of some kind.