Why can't I connect a dimmer in line with an outdoor security light?
Generally, schemes where you want to control a light 2 ways at once, do not work.  The two methods end up in contention/conflict with each other.
The motion sensor and photocell units need 24x7 power. They do not work on reduced power, partial power or the wacky waveform you get out of a triac dimmer, which is made only for lights. Since you are using a single light+motion sensor+photocell unit, you can't separate out and dim only the light, unless the internal arrangement of the unit allows you to do so, which is pretty rare these days.
Modern motion sensor+photocell+LED units generally have a single power supply on board that makes 12V, both for the motion sensor and the light.  (12V motion sensors are ~$10, 120V motion sensors are ~$20 due to the need for an internal power conversion and a higher safety rating switch).  If you could access the internals of the unit, it's certainly easy to get a PWM dimmer for ~12V LEDs, that could go between the sensor and the LED.  The cheapies have a physical knob that will stay where you put it.
Honestly, your best option - though it'll be a little complex - is to do exactly that with discrete components.  The photocell is generally part of the motion sensor, and so you don't need 2 components there.
So you have a 12V power supply... then a 12V motion sensor... then a 12V PWM dimmer... then 12V LED lights of your choice, which are readily available.
It'd be possible to do that same thing in 120V, however choosing a dimmer would be a challenge.  Most dimmers these days do not like being downline of a switch, and will do undesirable things like reset themselves to full brightness etc.