Changing some part of the paper after getting accepted for a conference
I am wondering whether I should modify the paper...based on the comments of the reviewers.


Yes you should. Acceptance was (most likely) coupled with an expectation that you'd address some or all of the comments. The reviewers (presumably) consider their comments relatively minor, since they'd have rejected otherwise, and are trusting you to make sufficient changes (that they won't check).