Being told a conflict of interest is acceptable for paper review
From an outsider's perspective, both choices are now ethically just. You've declared the potential conflict of interest and been assured that the authority in question doesn't see one ( hopefully you did this through a means which let you keep a record).

So now it pretty much comes down to whether giving the paper a fair review makes you feel uncomfortable. If it does, then get back to the committee as soon as possible, and tell them as clearly as possible that you can't do the review. Otherwise, review the paper, making full use of your deep understanding of the context of the work.

If you're wondering whether the committee member's statement is ethical, then consider whether you, as an obviously knowledgeable person in the field, could think of three other possible reviewers. Of course, if you can, then maybe one answer is to say no, but pass on those names instead.