Should I worry about a referee who recommended rejection being offended if my paper is accepted?
No, you should not worry.  Referees offer opinions.  The actual decision to accept or reject a paper rests with the editor.  The referee may very well be offended by the editor's decision to ignore her opinion, but that's certainly not your problem.