How to deal with sources whose authors I don't have good relationship with?
If you feel that the work is relevant, you must cite it - not citing would be unethical.

Also, most people are happy if they are cited - regardless of their relationship to the person who cites them. Citing someone does not mean liking someone - even when criticizing someone's work, you have to cite.

And if you can think of no particular reason why they should not want to be cited by you, there probably is no such reason.