Will addressing reviewers comment and accepting drawback of work take back my acceptance?
In general, if you got an acceptance notice from a conference, then you are accepted.  At this point, it's up to you just how much to take the reviewers into account in revising your paper. 

Personally, I do recommend changes in response to every issue raised, because that will generally make you paper stronger.  Even if a reviewer is dead wrong, it at least points to a place where readers can become confused. In this case, kudos to you for taking the reviewers seriously and thinking carefully about their feedback, rather than just being defensive.

Note that in some cases you may get a "conditional acceptance", and in that case you definitely have to be concerned about the reviewers' opinions of your new version.  That does not seem to be the case here based on what you have written, however.