Recessed lights still getting power when switch is off
It's a Lutron dimmer with night light.

First, that should never have been used to control an outlet. You must not put dimmers on outlets, because if someone plugs in a non-lamp, the power-shaping issues can cause equipment to dramatically overheat and start a fire.  If you have any more dimmers controlling receptacles, remove them now and fit standard switches with no powered features. 

Dimmers and night-lights are both features that require power themselves to operate.  Traditionally, powered switches like this got this power by leaking power through the light bulb.  On an incandescent bulb, at very low currents they remain unlit - in fact they are practically a dead short until they start glowing.  That is perfect for this.  

However, LEDs actually put the power to good use - even small, defective power like this.  

Pop out the switch and look for whether the switch takes a neutral wire.  If it does not, this is a bad choice of switch.  You need a more modern type, which requires neutral (I assume it's available there), powers itself between always-hot and neutral, and only sends power to the bulbs for lighting the bulbs. 

The dimmer "clicking hard off", that is a feature of many dimmers, but here, it is a red herring.  It still has a night light.