Added by Kevin Horton over 1 year ago. Updated about 1 year ago.
Bartosz - iocage has done that from the beginning :) The issue is the zfs mountpoints being transferred.
Sounds like this is fixed in the next replication incarnation so marking closed.
The data related to two of my three iocage jails has disappeared from /mnt/iocage, which makes iocage very unhappy. The jails are still seen in the output of jls, and jexec can be used to enter the jails. /mnt/iocage/jails did show all created jails a few hours ago, but two of the three disappeared at some point, without any obvious user action as a trigger.
drwxr-xr-x  3 root  wheel  3 Jan  1 17:54 .
drwxr-xr-x  8 root  wheel  9 Jan  1 17:53 ..
This is likely because those steps you mention are forcefully unmounting the jails root.
Nothing iocage does will be unmounting the jail. I suggest running zfs mount -a and figuring out why/what is causing these datasets to be unmounted.
"zfs mount -a" does not cause the missing jails to reappear. They are still missing from /mnt/iocage/jails, yet still shown by "jls" and still can be entered via "jexec"
What does `mount` show? I'm suspecting the pool may have done something funky and reimported and they are scattered. If they show in jls, you can always jail -r JID of the jail. That should allow you to troubleshoot without worrying about the processes in the jail running.
I restarted the system yesterday, as I had suddenly lost the ability to enter one of the jails using "jexec". I had rebooted twice before in an attempt to clear the issue, but those earlier reboots had not resolved it. This latest reboot seems to have resolved the problem, at least for the moment. The iocage jails have reappeared, and iocage is working normally.
"mount" shows the jails on the main pool mounted at /mnt/iocage, and the ones replicated to the backup pool are mounted at /mnt/backup/iocage. Earlier, when the problem was evident, the jails on the main and backup pools were both mounted at /mnt/iocage.
Note: I am aware of another user with the same or a similar problem. He may be able to provide information from a system that currently exhibits the issue.
Kevin: I'll close this out for now. If you or the other user can recreate, please attach a debug (System -> Advanced -> Save Debug) from the system to this ticket.
I get back the initial mounts for /mnt/iocage, and everything works fine. I have to repeat that workaround every hour at :00, since some process does that mount from backup/qandd/iocage to /mnt/iocage again. When I disable the replication task for the dataset carrying the iocage dataset to the backup volume, the shadowing mounts are not done any longer.
Screening this as the fix in iocage will be to avoid the traceback.
For those wondering how this is happening, iocage list actually parses jls, so the jails are "technically" running and available. But the mountpoint is shadowed, so every operation will fail. Considering iocage doesn't actually mount its dataset, but relies on it to be already mounted, this behavior isn't caused by iocage, as Heiko also describes seeing it without issuing iocage at all. But like Heiko also says, there is some other process that's causing these to shadow mount each other.
This will be assigned to Bartosz afterwards to investigate if it's the replication at fault.
One observation that may or may not be related: when I created my first jail using iocage, the system did not require a user input to specify which pool to use for the iocage jails. The system put the iocage jails on my backup pool, which was the first pool when sorted alphabetically. I desired that the jails be on the "main" pool, as that was the poll that was replicated to my second NAS. I then used "iocage activate main" to specify that the iocage jails should be on the "main" pool. I suspect that this resulted in both /mnt/backup/iocage and /mnt/main/iocage being mounted at /mnt/iocage.
That's exactly what I did also. The root cause for this problem is that "iocage activate" sets the zfs property "mountpoint" to "/mnt/iocage". That zfs mountpoint property is replicated from the iocage activated dataset (qandd) to the backup dataset, both pointing to /mnt/iocage, see the output of zfs get below.
To eventually solve this bug, the zfs mountpoint property should be excluded from replication to a local backup volume. This is currently not yet possible using the FreeNAS GUI (old version), since the "zfs recv" process has to be told to replace that property by option "-o", see e.g. https://docs.oracle.com/cd/E23824_01/html/821-1448/gbchx.html#gjonv how to do this. Not sure if this can be handled by FreeNAS, since I saw no field in the GUI where I can provide additional "zfs recv" options.
As far as iocage and mounting is concerned, the first activated pool is given the mount '/iocage'. Any additional pool is given the mount '/pool/iocage' to avoid conflicts like this. The duplicate checking behavior was too naive, and has also been fixed in https://github.com/iocage/iocage/commit/515541180c23faa4b8b45de8ebe058bcafb320f2.
Passing to Bartosz to make sure mountpoint conflictions on the replicated side are dealt with.
From the legacy/current replication point of view mentioned situation occurs when 'Delete stale snapshots on remote system'(follow_delete in code) is checked. ZFS properties are sent only when this option is enabled(I'm not fully aware of the intentions).
Unfortunately It's too late to play with a fragile autorepl code but I can address those issues in the new replication engine.
I'm using custom ZFS properties in the new replication code extensively to store metadata and I'm encouraging you to do the same with IOcage if it is possible.
Please contact me directly if I can do anything before 11.1-u2 to solve this problem.