How to fix a "heartbeat failure" in Docker Swarm?
This might not be the answer to your specific cross data-center-ip setup.

I occasionally run into one or more swarm-nodes being status: Down and Availability: Active. Having the  Status.Message: "heartbeat failure". This can happen after a reboot.

What helped was to stop the docker daemon, remove /var/lib/docker/swarm/worker/tasks.db and start the docker daemon again.

from:
https://github.com/moby/moby/issues/34827#issuecomment-457678500

sometimes it fixes itself: https://stackoverflow.com/a/54126180/2087704