New Switch for Outdoor Area Lights
I cannot answer your questions regarding the wiring and why the lights do not switch on and off. But clearly there is something misunderstood regarding what wires have to hook to what in order to allow this to switch correctly. With that said there are a few things that do require comment here.


After removal of the timer box from the wall you just have the Romex cables poking through the wall. You are going to have to install a proper electrical box in the wall to secure the Romex cables, contain the electrical connections and provide means to mount the switch.
The quality of wiring workmanship in both the old timer box and the connections showing for the new switch leave something to be desired. For example the wire nut does not even cover the insulation on all the wires as showing below. Throw away those hard plastic wire nuts and get some of the softer plastic wire nuts that have longer skirts to cover the wires better. 





The type of smart switch you are using is one designed to operate without a neutral connection and as a dimmer for indoor lighting. I would really recommend another type to use for exterior lighting which is the Lutron PD-6ANS-WH. This switch eliminates the dimming function and uses a neutral connection. This configuration of smart switch will provide you much more reliable on/off operation for a variety of load types (which could come into play as your lighting fixture calls for a specialty type bulb).