Directly contacting a referee of my paper
This should depend on the journal policy. Both you as an author and the reviewer are bound by the rules the journal has in place with respect to the publication process. While it sounds like you are meeting with the reviewer solely for the purpose of improving the paper but it may come across as going out of one's way to others. To be on the safe side, I would suggest you contact the journal editor before organizing such a meeting. With a sign-off from the editor, you can be sure that you are playing a fair game. I would also keep co-authors in the loop if any.