LED light is on when it should not be
What you have done is connected the LED lights in parallel to the switch. This way, when the switch is on, the LEDs are shorted out and power flows to the pot lights. When the switch is off, you end up effectively with the LEDs and the pot lights in series. Because the LEDs are so low power, the power flows through the pot lights without giving them enough power to visibly light.

Check your switch box to see if there is a neutral wire there (probably white if you’re in the US). If so, you can connect the LED driver between that and one side of the switch. If you don’t have a neutral wire there, you’ll need to run a cable from a pot light to the LED driver. Connect it in parallel with the pot light.

One more comment. I’m assuming that the LED driver board is not designed to be in a junction box. Doing what you’re doing is probably against the electrical code (at least in the US) and could be dangerous, especially if the board shorts out to the box. If this causes a fire, your insurance may have grounds to not pay, or at least to drop you afterwards.