Am I doing something wrong with this sanding job?
The short ends you pictured are the cross grains of the board. What I see is that you have used a too coarse sandpaper on these edges. This should be easy to fix. You need to step down the grit of the sandpaper. Any final sanding should be done with 220 or 320 paper. On these end details, use fine paper, break its (paper's) back so it is flexible, and take your time working it on these ends by hand. It may take a few minutes, but it will smooth out nicely.