MongoDB replica set over ssh tunnels
The following setup is possible, without using TLS.
Datacenter 1 is the primary datacenter. All clients connect only to this one.
Datacenter 2 is only for realtime backups via replication.
On-Site is also only for realtime backups via replication.
This setup is NOT meant for failover, only for disaster recovery.
Datacenter 1:
MongoDB

mongod --bind_ip 127.0.0.1,172.17.0.1 --replSet test --port 2000
--rest --httpinterface --logpath data/replica/logs/log.txt --dbpath data/replica/wiredTiger --directoryperdb --storageEngine wiredTiger
--wiredTigerDirectoryForIndexes --fork

127.0.0.1 is for SSH access, 172.17.0.1 is for access from Docker containers.
SSH Tunnels (this server is responsible for creating the tunnels to Datacenter 2):

autossh -f -N -L 2001:localhost:2001 mongodb@dc-2.example.com
autossh -f -N -R 2000:localhost:2000 mongodb@dc-2.example.com

Datacenter 2:
MongoDB

mongod --bind_ip 127.0.0.1 --port 2001 --replSet test --rest
--httpinterface --logpath data/replica/logs/log.txt --dbpath data/replica/wiredTiger --directoryperdb --storageEngine wiredTiger
--wiredTigerDirectoryForIndexes --fork

On-Site:
MongoDB

mongod --bind_ip 127.0.0.1 --port 2002 --replSet test --rest
--httpinterface --logpath data/replica/logs/log.txt --dbpath data/replica/wiredTiger --directoryperdb --storageEngine wiredTiger
--wiredTigerDirectoryForIndexes --fork

SSH Tunnels (On-Site is responsible for creating the tunnels to Datacenter 1, no persistent connection exists between Datacenter 2 and On-Site)

autossh -f -N -R 2002:localhost:2002 mongodb@dc-1.example.com
autossh -f -N -L 2000:localhost:2000 mongodb@dc-1.example.com


The server in Datacenter 1 contains all the data which is to be replicated, Datacenter 2 and On-Site are empty databases.
On the machine in Datacenter 1 I enter mongo --port 2000, then I issue a
rs.initiate(
   {
      _id: &quot;test&quot;,
      version: 1,
      members: [
         { _id: 0, host : &quot;localhost:2000&quot;, priority: 1, votes: 1 },
         { _id: 1, host : &quot;localhost:2001&quot;, priority: 0, votes: 0, hidden: true },
         { _id: 2, host : &quot;localhost:2002&quot;, priority: 0, votes: 0, hidden: true }
      ]
   }
)

I guess priority: 0, votes: 0, hidden: true on the secondaries is optional here, it does what I want, just have the two secondaries be a part as silent, invisible data collectors for backup purposes. I'm not sure if changing this would have any side-effects as it could be possible that then a connection between both secondaries should be available, which it is not, in this case (they can't see each other).
I could use a slaveDelay on one of the hidden members in order to make it lag for let's say half a day, but since I will be making plenty of backups from both hidden databases via mongodump, it's not really neccessary.
In order to be able to connect to the secondaries via mongodb tools I needed to issue a rs.slaveOk()
Nothing had to be changed in the clients, they connect to 172.17.0.1:2000.
I tested a reboot of the machine in Datacenter 2 and during the reboot I inserted new items in the DB, after about 30 seconds the rebooted machine synced to Datacenter 1 as expected.
Datacenter 1 has a fast server with lots of RAM and CPU and only a little bit of SSD storage, Datacenter 2 has a little bit of RAM but lots of HDD, so this seems to be a good solution.