cident of timing—such as scheduling or speed of message delivery—that could be reversed without any effect on the rest of the execution.) When do hosts communicate with one an

other? What is the minimal set of events that may be responsible for a particular outcome?  Querying for interaction patterns between hosts. The distributed system is designed to perform certain patterns of interaction. When do these occur? When do they begin to occur but are interrupted and fail to complete? What patterns exist that may not be documented or may be an emergent property of the system as a whole?  Identifying structural similarities and differences between pairs of executions. Given a reference implementation and a buggy implementation, how and when does their behavior differ? Given a single system, characterize the differences between faulty and non

faulty executions. How does the system react to differing environments, such as hardware failure? What are the runtime differences between two algorithms designed to serve the same purpose?

Understanding the relative ordering of events. Given two events at different hosts, is one potentially causally dependent on the other? (If not, then their relative order is an ac- cident of timing—such as scheduling or speed of message delivery—that could be reversed without any effect on the rest of the execution.) When do hosts communicate with one an- other? What is the minimal set of events that may be responsible for a particular outcome? Querying for interaction patterns between hosts. The distributed system is designed to perform certain patterns of interaction. When do these occur? When do they begin to occur but are interrupted and fail to complete? What patterns exist that may not be documented or may be an emergent property of the system as a whole?

Identifying structural similarities and differences between pairs of executions. Given a reference implementation and a buggy implementation, how and when does their behavior differ? Given a single system, characterize the differences between faulty and non- faulty executions. How does the system react to differing environments, such as hardware failure? What are the runtime differences between two algorithms designed to serve the

We have developed a novel method for logging and analyzing distributed systems to address these challenges of understanding, querying, and comparing distributed system executions. Our method consists of two parts, ShiViz and XVector. ShiViz visualizes a distributed system execu- tion as a time-space diagram [66] that explicitly represents the happens-before relation between events (for examples, see Figures 1 and 2). ShiViz contains features to support the following system

 Event ordering. The ShiViz time-space diagram explicitly but compactly represents the relative ordering of events across hosts in the system, capturing concurrency among events. The visualization also links the time-space diagram to the corresponding textual entries in the log. ShiViz enables the developer to simplify the graph by transforming it to elide information that is not relevant to their current task.  Interaction patterns. ShiViz allows developers to construct event sub-graphs and search for them in the time-space diagram. These sub-graphs may include constraints on host identifiers and event meta-data.

Event ordering. The Sh

iz time-space diagram explicitly but compactly represents relative ordering of events across hosts in the system, capturing concurrency among The visualization also links the time-space diagram to the corresponding textual in the log. Sh

iz enables the developer to simplify the graph by transforming it to information that is not relevant to their current task.  Interaction patterns. Sh

iz allows developers to construct event sub-graphs and for them in the time-space diagram. These sub-graphs may include constraints on identifiers and event meta-data.

ACM Transactions on Software Engineering and Methodology, Vol. 29, No. 2, Article 9. Pub. date: March 2020.

 Multiple execution comparison. ShiViz can present two execution graphs side-by-side to help developers compare these executions. ShiViz includes algorithms to highlight dif- ferences between pairs of executions and supports the clustering of executions based on features.

ShiViz requires the distributed system to generate a particular kind of log, which captures con- currency information. XVector, a suite of libraries for C, C++, Java, and Go, helps automate this process.1 XVector interposes on communication and logging channels at each host in the system to add vector clock timestamps [36, 80]. These vector timestamps capture the happens-before re- lation [68]. XVector produces a textual log of printf-style messages augmented with vector clock timestamps. ShiViz reads these logs to reconstruct the graph of inter-host communication and display a time-space diagram. ShiViz includes specific capabilities to help developers implement correct systems. The capabilities help a developer understand event ordering, query for interaction patterns, and compare executions.

We evaluated the behavior-understanding capabilities in ShiViz through three studies:

(

) We ran a controlled experiment with a mix of

undergraduate and graduate One group of participants studied distributed system executions using Sh

iz and group without Sh

iz. The study asked all participants to answer questions about system represented by the executions. (

)

students in a distributed systems course used Sh

iz as part of two homework ments to help them debug and understand their implementations. (

) We ran a case study with two systems researchers who were developing complex tributed systems to evaluate the end-to-end usefulness of Sh

iz to developers in work. Across these studies, we collected the developers’ impressions via surveys