How much damage would I be incurring if I turn down an informal postdoc offer?
Since you haven't  agreed to take the job, there's no damage being done at all.  Being offered a job does not put you under any obligation to take it, in academia or otherwise.  You shouldn't feel embarrassed or awkward.  If you're sure about your decision, then it's helpful to tell the professor now.