Why ride a fixed-gear bike?
Your understanding is correct. 

Why do people ride them? Some random answers:


Maintenance is very, very low. You have to keep the tires and chain in good working order and, on a bike you ride on the road, hopefully some form of a brake. That's it.
On most velodromes, you have to ride a fixed gear bike, so if you race track, you have no choice.
There is something to be said for how a fixed gear will force you to develop a smoother pedaling style since you simply cannot stop. Due to this, they are somewhat popular for 'off season' training by serious road cyclists.
(Hesitating to mention this...) There is a certain segment of the population that loves retro and simple things. In addition to being possibly the most efficient people moving device that exists, the fixed gear bicycle can be an important fashion accessory.


Hard to get started/Hard to go fast?

Yes. Generally someone who has a road worthy fixie will have selected a gear that works well for the terrain and speed that they like to ride at. I've only ridden fixed gear bikes on the track, but I'd imagine for a city fixie, you'd optimize for a fairly low speed but someone who takes these more seriously can answer better.