How do I repair screw holes in a brick wall?
There are epoxy repair systems for wood. It should work here too. Once it hardens, the epoxy is a tough, sandable, paintable medium that can de drilled. Make sure it keys properly into the hole, so that it does not just pull free under load. This may involve an undercut if necessary.

In fact, I just checked, and there are epoxies available specifically for masonry repair too, but the wood product should be sufficient here, and is easily found in most home stores.

An alternative might be to add a flat wood molding behind the entire length of rod. 3/8 of an inch thick should be sufficient. Paint to match the wall. Now anchor the curtain rod to the wood, using expanding drywall anchors, if necessary.