Best cost efficient plug-in heating devices / radiators
Electrical baseboard heaters and plug-in resistance heaters are already 100% efficient, in that 100% of the electrical energy that goes into them is turned into heat. Whether it's a ceramic heater, oil-filled radiator, or radiant heater, they all are 100% efficient.

However, as you found out, even at 100% efficiency, electrical resistance heating is expensive.

This is why many people that need to use electrical heating will use a heat pump, which can actually be more than 100% efficient because instead of creating heat directly from the electricity, they pump heat from outside the house to inside the house. But this is not a simple retrofit, so you can't simply replace your baseboard heaters with a central heat pump system (though if you live in a relatively mild climate, you can find window unit heatpumps that might be more efficient than a baseboard heater)

One thing that might help would be to turn down the baseboard heaters, and use radiant heaters in places where you want targeted warmth. For example, if when you're in the livingroom you spend most of the time on the couch, you could turn down the thermostat in that room and use a radiant heater in front of the couch (but a safe distance away) to keep you warm even if the rest of the room is not comfortably warm.