Does drywall "wear out"?
No, drywall does not wear out. But holding excessive loads on anchors in drywall is a guarantee of failure. Drywall anchors do tolerably well in shear (loads downward, like hanging a picture) and fairly poorly when yanked on/outwards repeatedly. Once they move at all, they will quickly crumble until they come right out. Nor are they at all good for anything more than a small, lightly loaded shelf, and dubious at that.

If there's no stud where you need to support a troublesome load, and especially if you are already looking at repainting due to needed drywall repair, open the wall up more and put blocking between the studs to support the loads, then repair the larger hole by standard drywall repair methods.