Can it be unfair when the same referee reviews a manuscript again at a different journal?
If you have concerns that the reviewer isn't being fair or just, you can always contact the editor and ask for a new reviewer. It is not entirely uncommon, I know of researchers that have a "blacklist" of people that they do not allow to review their works.

But consider that the reviewer might have a point. I know it is not your case, but I get really disappointed when I get the same paper twice and it is exactly the same text. Regardless if is the same process or not, a proper review would most likely improve the paper, so to see that not even the typos got fixed (in my case) is hard.

More to the point, how did you change your paper to make sure nobody else reading it would reach the same conclusions as this reviewer?