In my most recent previoust blog post I showed some perfmon graphs, with 1 second collection interval, from some tests I was running with DiskSpd in an 8 vcpu VMware VM, against a guest LUN managed by the pvscsi vHBA.
In that particular test run, the main concern was the high privileged time on vCPU 0, which is also the location of the SQL Server logwriter.
Although most of my tests showed high CPU utilization on vcpu 0, there was some variability. Sometimes it would be a different single vcpu bearing the cost of handling the vHBA. Sometimes the work would be shared among a few vcpus.
Here are some results of a separate test, mere minutes away from the last test I shared, on the same system. These results show another concern I have about high SQL Server disk IO within VMware VMs.
In this case, cost of managing disk IO has been balanced across vCPUs 0, 1, 2, and 3. The remaining 4 vcpus have little involvement in managing the disk IO.
That presents an interesting question - why is the cost dispersed in this case? It may be extremely preferable to disperse the cost over 4 vcpus, rather than keep vcpu 0 (*especially* vcpu 0) nearly fully consumed with privileged time.
But there's another important question - and one which may be familiar to lots of other folks running SQL Server in VMware VMs by the time I'm done. What's going on in the nearly 6 second valley?
If I look at IOPs (reads/sec since this was a read-only test) the valley is evident, too.
Sometimes a chart of the values over time put performance divots with extreme outlier values into better perspective.
So - average read latency of under a millisecond at roughly 60,000 read IOPs with a divot to under 300 read IOPs at average read latency from 96 milleseconds to over 1 full second. And one second ending at 14:50:24 with NO reads completed at all.