Is it possible to design this circuit with a single potentiometer (L-pad)?
Normally reactive components such as inductors and capacitors are used to provide attenuation within/outside of specific frequency ranges and this by definition is how cross-overs are built.

Reactive components are favorable because they do not dissipate power apart from minimal R losses.

If you want to attenuate the signal being driven into a power load such as a speaker then I believe you will need a fairly robust potentiometer. If you are delivering 40 watts into a speaker, and you want to attenuate it by -3dB, then your potentiometer will need to be 8ohms and capable of handling 20 watts. Something the size of a high school lab rheostat would be suitable, but impractical and slightly inductive. I would suggest you consider attenuation of the signal entering your amplifier rather than the signal coming out of it.

If however you're dealing with 250mW headphone speakers then the potentiometer will work, however you may want to consider a Logarithmic type as the parallel impedance equation is not linear with differences in R.