What is the problem with my switching circuit?
This is an excellent example of diodes detecting stray common mode CM E-fields and conducting current by rectification in the LEDs.
The unbalanced switch resistance causes this common mode field.
The solution is to reduce CM EMI by twisted pars then raise the CM impedance above ~1MHz for the fast rise times with a ferrite CM choke or BALUN. These are needed on all high speed high current switched wires.
It can be a clam-shell ferrite like those used in all VGA cables which is moulded in plastic with no air gap or a large torroid with all wires wrapped thru the torroid as many times as possible.

Start by adding a wire from ground and wrap around each cable from source and terminate with a low ESR capacitor on LED boards.  Since you have no ground on the LED boards, adding this cap and ground wire, reduces the loop area of the offending board circulating current but will easily couple within the same distance gap as the size of the board loop.
Shielding or connection from an earth ground will also help absorb stray switched E fields greatly by lowering the impedance of your floating battery ground or a sheet of alum. foil connected to earth ground also works.