How should I fill the holes where my curtain rod was, so that I can reuse them?
I had the same issue with my curtains.  I was able to use the Tapcon concrete 3/16” by 2-1/4” concrete anchors.  

If your holes are too large for the Tapcons: try drilling out the hole to a ½ inch in diameter and about two inches deep.  Then add some construction glue and pound in a ½ inch dowel.  You can then reattach the curtain hangers with wood screws.