12' sliding door frame shimmed up against header and sagging at point where shimmed
Without a full mental picture, here are my initial suggestions...
Can I remove those shims?
Probably. Head jamb anchor points are sometimes needed to prevent horizontal (in-out) movement in the frame, but if you have mounting flanges or something else on the door to do that they aren't needed. Most doors I've installed have been done without head jamb shims for just this reason.
How do I remove them?
A reciprocating saw (Sawzall) will cut the nails or screws and leave a bit more space to remove them. A flatbar as a lever can also relieve pressure.
Should I be worried about the sag?
You haven't told us what the header is, nor how much sag you're seeing. Chances are it has sagged all it's going to sag, unless the header was drastically undersized. Do you see evidence of sag above the header? If that's substantial and growing you should get an expert on site.