What's wrong with my UV map?
It is some sort of a uv stretching problem. You can find where it is by enabling "Stretch" in the uv editor:



Blue is what you want, green is okay, and orange/red is really bad.

My guess is that your problem is in this region, but I can't say this for sure unless you post your blend:



Edit:

The problem is that those are N-gons on the side. You need to remove all of the N-gons to unwrap successfully. I just triangulated, then removed the edge cutting across the center to make it a quad:



Then unwrap, and voila.

Here is the blend: