How to fix over-tightened metal drywall anchor?
If the anchor has been overtightened and is now loose in the wall, it cannot be salvaged. The plaster in the wall (or the drywall) has been pulverized and will not hold this anchor or any other.

You need to remove the anchor and the screw.  

AMENDED ANSWER

Based on new information from the Original Poster, the anchor is a molly type that mushrooms behind the drywall.  The best way to remove these is to push them in rather than pull them out. 

Insert an awl or a very narrow screwdriver alongside the anchor, pushing toward the hollow in the wall. Do this repeatedly around the circumference. You are basically crushing the little bit of plaster that is hold the collar of the anchor on the surface of the wall. You shoUld then be able to gently push the anchor into the wall hollow where it will drop into the bottom if the cavity.

If you have not damaged the area too much, you may be able to replace the drywall anchor with a strap toggle type anchor.



The legs on this anchor slide to move the crossbar parallel to them. The crossbar is slid into the hole and the legs are then adjusted so that the crossbar is now perpendicular (but parallel to the inside surface of the wall. The front plate on the toggle is then slid tight to the wall and the extra legs are snapped off. The hanging item is then secured by a bolt that goes through the crossbar.

If the hole from the original anchor is much bigger than the toggle diameter, you probably need to patch the hole and move to a different, sound section of the wall and start again.

This answer discusses a range of mounting methods.

In general, I try to avoid molly type anchors because they are prone to the very problem you experienced.  

 The links and images are for illustration purposes only and are not endorsements