Atmel ATxmega128A identical boards - one fails to enter programming mode
Apparently your particular design implementation (PCB layout, traces, components, pullups, etc) is marginal for this interface to work at 4MHz. Given the spectrum of manufacturing tolerances, some samples are lucky to work at 4MHz, some not. It is likely that if you manage to have, say, 3.5MHz clock, both boards will work fine.