How long should concrete set for a post before I perform additional work?
The 28 days mantra is a bit misleading. Concrete cure times vary widely depending on environmental factors and mix. It's often at 80% or more by a week. 

That said, we don't know much about your pour. If you have 4" of concrete around the post, it's fairly fragile and drilling could crack it. If you  have 8 or 12", it's much more robust and would probably tolerate drilling just fine, if you keep pressure light. All this depends on adequate moisture having been provided and temperature maintained. 

Add more information for more detail in answers.