回答 vs. 回复 vs. 回应 vs.答复： What is proper usage?
回答often used when somebody answer your question, usually face to face.
回复often used when they reply you via text or phone.
回应is used when you done something, others do another thing in respond.
答复is used when you ask question or have some petition to the government or other big agency, you will get a respond from them.  

In your example,


  "I'm so sorry I took so long to respond to your message!"  


should use 回复


  "‘Of course!' the man replied!"


should use 回答


  "I waited for a long time, but still no response"


should use 答复

as for


  "He was not happy with my reply"  


it will mainly rely on the context you are in,