Poorly installed drywall anchors (by me)
There are many types of anchors. I suspect you used the basic cheap plastic plugs. I'd look to the various expanding or toggling types and simply use the same holes. 

Otherwise, get some setting type joint compound and press it in firmly. Knife it off and allow it to cure thoroughly before a light sanding.