Should I accept an invitation to give a talk from someone who might review my proposal?
It's the potential reviewer's responsibility to declare or avoid any conflict of interest, not yours.  

If the potential reviewer isn't asked to review your proposal (or declines for other reasons) then there's no conflict of interest.  

If the potential reviewer does end up reviewing your proposal and you've declined an invitation to speak at his institution that potential reviewer might hold it against you.

If you do accept the invitation, this wouldn't by itself be sufficient to create a conflict of interest that would prevent the potential reviewer from reviewing your proposal.    

Thus I wouldn't hesitate to accept the invitation.