Do the wattage from connected devices need to add up with the given wattage from a step-down converter?
Does it mean that I can connect multiple devices not producing more than 36W power?


It means you can connect multiple devices not demanding more than 36 W of power in total.

Some switch-mode converters may behave badly (be more noisy, or allow output voltage to drift up) if the load is much lighter than they are designed for. If your load is less than, say, 25% of the designed capability of the converter, you should check on the performance with light load before selecting that converter.