docs: snapshots: block devices not explicitly flushed
@@ -11,7 +11,14 @@ guest workload at that particular point in time.
A Firecracker microVM snapshot can be used for loading it later in a different
Firecracker process, and the original guest workload is being simply resumed.
-It is not guaranteed though that the state of the network connections survives
+
+The original guest which the snapshot is created from, should see no side
+effects from this process (other than the latency introduced by the snapshot
+creation process).
+
+Both network and vsock packet loss can be expected on guests that are resumed
+from snapshots in another Firecracker process.
+It is also not guaranteed that the state of the network connections survives
the process.
In order to make restoring possible, Firecracker snapshots save the full state
@@ -51,7 +58,7 @@ resumed microVM.
is not enabled, results in the full contents of guest memory being written to the
snapshot.
- The _memory file_ and _microVM state file_ are generated by Firecracker on snapshot
- creation, and the disk contents are flushed to their backing files.
+ creation. The disk contents are _not_ explicitely flushed to their backing files.
- The API calls exposing the snapshotting functionality have clear **Prerequisites**
that describe the requirements on when/how they should be used.
@@ -131,9 +138,12 @@ Details about the required and optional fields can be found in the
- The file indicated by `snapshot_path` (e.g. `/path/to/snapshot_file`) contains the
devices' model state and emulation state. The one indicated by `mem_file_path`
(e.g. `/path/to/mem_file`) contains a full copy of the guest memory.
- - The block devices contents are flushed to their backing files. At this point, these
- should be backed up externally by the user. The generated snapshot files are
- immediately available to be used (current process releases ownership).
+ - The generated snapshot files are immediately available to be used (current process
+ releases ownership). At this point, the block devices backing files should be
+ backed up externally by the user.
+ Please note that block device contents are only guaranteed to be committed/flushed
+ to the host FS, but not necessarily to the underlying persistent storage
+ (could still live in host FS cache).
- If diff snapshots were enabled, the snapshot creation resets then the dirtied page
bitmap and marks all pages clean (from a diff snapshot point of view).
@@ -313,3 +323,20 @@ properties are preserved and guaranteed before resuming the workload.
We've started a discussion on how the Linux operating system might securely
handle being snapshotted [here](https://lkml.org/lkml/2020/10/16/629).
+
+## Known Issues
+
+### Vsock must be inactive during snapshot
+
+Vsock device can break if snapshotted while having active connections.
+Firecracker snapshots do not capture any inflight network or vsock (through the
+linux unix domain socket backend) traffic that has left or not yet entered
+Firecracker.
+
+The above, coupled with the fact that Vsock control protocol is not resilient
+to vsock packet loss leads to Vsock device breakage when doing a snapshot while
+there are active Vsock connections.
+
+#### Workaround
+
+Close all active Vsock connections prior to snapshotting the VM.
