Best way to replace caster wheels
Best option - drill hole out to 3/4", cover a 3/4" wooden dowel with wood glue, stuff it in the hole, wait for glue to dry, cut it off flush, drill new hole.

Wood filler is not a quality repair you should expect to last.

You might have enough room to drill your small hole beyond the current hole.