Our XVector vector clock instrumentation libraries have (1) a network overhead due to additional vector clock timestamps in network payloads, and (2) a vector clock logging overhead. We evaluate both of these overheads in the GoVector library, which is an XVector library for Go lang. Our experience using other XVector libraries in teaching and with open-source systems indicates that other XVector libraries have similar overheads. We use GoVector to instrument etcd, a popular distributed key-value store that uses the Raft consensus protocol [87] to achieve the Experimental setup. We ran all experiments on an Intel machine with a 4-core i5 CPU and 8 GB of memory. The machine was running Ubuntu 14.04, and all applications were compiled using Go 1.6 for Linux/amd64. We exercised etcd by loading it using the Yahoo! Cloud Serving Benchmark (YCSB-B) with a uniform distribution (50% gets and 50% puts on random keys). The etcd Raft cluster consisted of three replication nodes. We measured the latency and goodput on both the uninstrumented and the instrumented versions of etcd. In both experiments, the maximum number of clients that could execute concurrently without reaching maximum CPU utilization was 72.

Identity and the other are the node's logical clock timestamp. The overhead of vector clocks is fore a product of the number of nodes interacting during execution and the number of 64-bit node's messages. In practice, adding vector clocks to messages slows down each, which can impact the behavior of the system. Figure 11 shows the goodput for uninstrumented and instrumented Raft with each point representing an average over three executions. Adding vector clocks to Raft slowed down the broadcast of heartbeat messages and caused an increase in goodput with 36 clients. As the number of clients grew, the size of the vector clocks overcame this goodput saving. Across these experiments, goodput decreased by 13% on average, and the maximum decrease was 23%.

GoVector imposes a latency overhead in computing vector clocks and in serializing and deserializing messages with vector clocks. Figure 12 shows Raft's latency processing client requests in the uninstrumented and instrumented versions of Raft. Across these experiments, the latency increase was 7% on average with a maximum increase of 21%.

In the above experiment, we also collected information on logging overhead. We found that the average execution time of a single logging statement is 20 microseconds. In our local area network with a round trip time of 0.05 milliseconds while running etcd with 1-second timeouts, we were able to execute approximately 50K logging statements per node before perturbing the system.

The overhead imposed by XVector makes it a viable tool during development but not in production. We believe that existing distributed tracing systems, such as X-Trace [39] and Dapper [110], which are intended for production use, can be extended with tracking of vector-clock timestamps. We have also successfully reconstructed ShiViz-compatible logs from X-Trace traces.

ShiViz is designed to run in the browser, making it more accessible and easier to use. However, design choice does impact its ability to scale to large and complex logs. In this section, we report scalability results from using ShiViz on a variety of logs. In each experiment, we report the time to load and visualize the log with ShiViz in a browser. In every logUsable and reasonably performant tools are necessary for complex distributed systems. The XVector and ShiViz tools have been introduced in a practitioner-oriented article [14] and have been used by research groups, developers, and companies. ShiViz is employed in research projects for debugging distributed systems [82, 107], and it is also used by P and P# projects within Microsoft to visualize traces of executions during debugging.13 Akka is a popular toolkit for building concurrent and distributed systems using actor-based programming in Java and Scala. Akka includes scripts for developers to convert their logs into ShiViz format.14 TLC is a model checker for TLA+ and PlusCal modeling languages that supports ShiViz for visualizing the counter-example trace from a model checking run of a concurrent model.15 The Bro network monitor also includes ShiViz support to help developers visualize distributed network threats.

THREATS TO VALIDITY
Salman et al. [102] and Pham et al. [89] have demonstrated that conclusions based on studies using students can generalize to the broader developer community. We evaluated ShiViz with 70 participants in an undergraduate distributed systems course with fourth-year CS majors at the University of British Columbia, a major research university (Section 6.2.1). Most of these became novice professional software developers only a few months later; we expect our results to generalize at least to novice engineers and perhaps beyond. 

While the goal of our ShiViz evaluations was to evaluate a set of user interface features and underlying algorithms, such evaluations and interface design are inherently iterative. The developers' interactions with the tool inform the tool makers and suggest new interface features. For example, our studies revealed that developers might benefit from highlighting differences between two executions, as described in Section 4.3.1. We elected to implement this feature as part of ShiViz and include the description of this finding here, even though the ShiViz version used for evaluation did not yet include this feature. ShiViz displayed executions side-by-side but did not highlight differences. Adding such features could, in theory, alter our results, although the effect is unlikely.