10https://github.com/MPI-SWS/xtrace-shiviz/.



Fig. 13. ShiViz scalability for logs with varying number of total events.



14. ShiViz scalability for logs with varying number of hosts.



Fig. 15. ShiViz scalability for logs with varying percentage of communication events.

were communication events), and each log had between 2K and 24K total events. Figure 13 shows the time to visualize each of these logs with ShiViz. The plot illustrates that ShiViz has a nonlinear cost for processing logs in terms of the number of log events. Note, however, that ShiViz visualizes logs with fewer than 5K events in under 10 seconds. This accommodates many systems: Distributed tracing logs from the microservices-based system below contain as few as a hundred events.

In the second experiment, we varied the number of hosts in the log. Each log in this experiment had 12K events with 50% being communication events, and varied the number of hosts from 2 to 24 in steps of 2. Figure 14 shows the visualization time with ShiViz for these logs, which is around 22 seconds for all the logs. This indicates that ShiViz is not impacted by the number of hosts in the log, vector clock length (which is proportional to the number of hosts), and the number of visual host timelines that ShiViz has to generate for logs with more hosts.

In the third experiment, we varied the fraction of events in the log that were communication events. Each log in this experiment had 12K events split evenly across 12 hosts, but the fraction of communication events varied from 10% (mostly local events) to 100% (all events are communication events). Figure 15 shows the time to visualize these logs with ShiViz. The time is relatively stable,

ACM Transactions on Software Engineering and Methodology, Vol. 29, No. 2, Article 9. Pub. date: March



Fig. 16. Description of different types of interactions with the social network microservices-based applica- tion from the DeathStarBench benchmark suite [42] and the properties of the log produced by distributed tracing each of these interactions.

as the fraction of communication events varies. The cost of an extra communication edge is (1) the computation to determine which two log events to connect, and (2) an extra communication edge visual element.

The above three experiments were on synthetic logs to help us benchmark ShiViz. We also considered real logs from DeathStarBench, a recently published microservices benchmark suite by Gan et al. [42]. We used the social network system in this suite. It is composed of 36 microservices and has over 60K LOC in a variety of languages.

We used the built-in social network user interactions, such as register for a new account, this application to generate the workload. We then captured the distributed trace caused by interaction across the microservices by using an instrumented version of the benchmark X-trace distributed tracing [39].11 We then converted the collected distributed trace into a compatible log and visualized it with ShiViz.

Figure 16 describes the benchmark interactions we used to generate the logs, the size of each log, the number of microservices in each log, the time ShiViz took to visualize the log, and the

Figure 16 shows that the logs had fewer than 400 events from 2–22 microservices. ShiViz visu- alized each of these logs in under a second and collapsed at least 70% of the local events in these traces.12 Based on this experience, we conclude that tracing specific parts of a complex system, such as a specific interaction in a social network system, is the most scalable way of using ShiViz.

Evaluation Summary

Overall, our evaluation suggests that ShiViz is easy to learn and to use. In particular, we found evidence to answer three of our research questions, RQ1–RQ3, in the affirmative. We also found that ShiViz is useful in a variety of contexts, including confirming system behavior, debugging, identifying anomalous behavior, and supporting system comprehension. The overhead of vector clock instrumentation depends on the system and its communication patterns; however, our ex- periments with etcd Raft indicate that our existing instrumentation libraries are usable and have reasonable performance in the context of complex distributed systems.

Since describing the XVector and ShiViz tools in a practitioner-oriented article [14], these tools have been used by other research groups and developers and researchers at several companies. For example, ShiViz is used in research projects focused on debugging of distributed systems that

11https://github.com/JonathanMace/DeathStarBench.

Visualizing Distributed System Executions

use ShiViz to visualize the traces they produce [82, 107]. ShiViz is also used by P and P# projects within Microsoft to visualize traces of executions during debugging.13 Akka is a popular toolkit for building concurrent and distributed systems by using actor-based programming in Java and Scala. Akka includes scripts for developers to convert their logs into ShiViz format.14 TLC is a model checker for TLA+ and PlusCal modeling languages. TLC supports ShiViz for visualizing the counter-example trace from a model checking run of a concurrent model.15 As a final example, the Bro network monitor includes ShiViz support to help developers visualize distributed network

THREATS TO VALIDITY

Salman et al. [102] and Pham et al. [89] have demonstrated that conclusions based on that use students can generalize to the broader developer community. We evaluated ShiViz 70 participants in an undergraduate distributed systems course with fourth-year CS majors at University of British Columbia, a major research university (Section 6.2.1). Most of these became novice professional software developers only a few months later; we expect our results generalize at least to novice engineers and perhaps beyond.