How to invite potential collaborators for personal visit?
With apologies, and a firm understanding that lack of funding may be a dealbreaker.

It would certainly help if you can cover part of the expenses: "If you are ever in [Country X], I hope you will also visit [my institution] and we would be happy to pay for your hotel [or else do you have a couch?] and pay for around [small number] towards your travel expenses. You are of course very welcome to make a special trip but I regret that funding is tight and we would be unable to pay your international travel expenses."

... or something like that.