block: add perf docs for the Async io_engine
@@ -51,6 +51,56 @@ If a block device is configured with the `Async` io_engine on a host kernel
older than 5.10.51, the API call will return a 400 Bad Request, with a
suggestive error message.
+## Performance considerations
+
+The performance is strictly tied to the host kernel version. The gathered data
+may not be relevant for modified/newer kernels than 5.10.
+
+### Device creation
+
+When using the `Async` variant, there is added latency on device creation (up
+to ~110 ms), caused by the extra io_uring system calls performed by
+Firecracker.
+This translates to higher latencies on either of these operations:
+
+- API call duration for block device config
+- Boot time for VMs started via JSON config files
+- Snapshot restore time
+
+For use-cases where the lowest latency on the aforementioned operations is
+desired, it is recommended to use the `Sync` IO engine.
+
+### Block IOPS and efficiency
+
+The `Async` engine performance potential is showcased when the block device
+backing files are placed on a physical disk that supports efficient parallel
+execution of requests, like an NVME drive.
+It's also recommended to evenly distribute the backing files across the
+available drives of a host, to limit contention in high-density scenarios.
+
+The performance measurements we've done were made on NVME drives, and we've
+discovered that:
+
+For __read__ workloads which operate on data that is not present in the
+host page cache, the performance improvement for `Async` is about 1.5x-3x in
+overall efficiency (IOPS per CPU load) and up to 30x in total IOPS.
+
+For __write__ workloads, the `Async` engine brings an improvement of about
+20-45% in total IOPS but performs worse than the `Sync` engine in total
+efficiency (IOPS per CPU load).
+This means that while Firecracker will achieve better performance, it will be
+at the cost of consuming more CPU for the kernel workers. In this case, the VMM
+cpu load is also reduced, which should translate into performance increase in
+hybrid workloads (block+net+vsock).
+
+Whether or not using the `Async` engine is a good idea performance-wise depends
+on the workloads and the amount of spare CPU available on a host.
+According to our NVME experiments, io_uring will always bring performance
+improvements (granted that there are enough available CPU resources).
+
+It is recommended that users perform some tests with examples of expected
+workloads and measure the efficiency as (IOPS/CPU load).
+
## Developer preview status
View the [release policy](../RELEASE_POLICY.md) for information about developer
@@ -98,20 +148,3 @@ kernel including mitigations for the aforementioned mitigations is released and
support for it is added in Firecracker.
Read more about Firecracker's [kernel support policy](../kernel-policy.md).
-
-## Recommendations
-
-In order to get the higher disk IO throughput, it is recommended to use the
-`Async` io_engine on hosts that support it.
-
-When using the `Async` variant, there is added latency on device creation (up
-to ~110 ms), caused by the extra io_uring system calls performed by
-Firecracker.
-This translates to higher latencies on either of these operations:
-
-- API call duration for block device config
-- Boot time for VMs started via JSON config files
-- Snapshot restore time
-
-For use-cases where the lowest latency on the aforementioned operations is
-desired, it is recommended to use the `Sync` IO engine.
