Toasting bagel in my oven
A toaster is more efficient than a toaster oven, which in turn is far more efficient than heating an entire conventional oven. 

Toaster statistics:
Approximate time to make toast – 4 minutes
Watt measured while running – 820 W
Resulting electricity usage – 0.055 kWh

vs 

Toaster oven statistics:
Approximate time to make toast – 5 minutes
Watt measured while running – 1110 W
Resulting kWh usage – 0.093

Lower kWh usage means the toaster is more efficient than a toaster oven.

Here is the source for the comparison of toaster vs toaster oven.

Now, you can compare the less efficient toaster oven to a conventional oven here (or to a microwave, but you don't really use that for toast). Toaster ovens use about 1/3 to 1/2 the energy of a conventional oven. So if a toaster uses less than a toaster oven, and a toaster oven uses far less than an oven, then clearly a toaster would be far less wasteful than the oven.

So YES, buy a toaster!