Across these studies, we collected the developers' impressions via surveys. Our evaluation results demonstrate that ShiViz supports both novice and advanced developers in distributed system development tasks. For example, our controlled experiment with 39 participants showed that those using ShiViz answered statistically significantly more distributed system understanding questions correctly than control-group participants without ShiViz, with a very large effect size. The two case studies provide qualitative data about the ShiViz developer experience, indicating that ShiViz helped these participants solve their problems faster than if they were to use other tools.

This article's main research contributions are:
1. A new method (and the supporting open source implementation for systems written in C, C++, Java, and Go, as detailed below) for logging and analyzing distributed systems to support common system understanding developer tasks.
2. Advanced, composable graph transformations for manipulating distributed system execution graphs, including constrained custom structured search and filtering by process.
3. A mechanism for side-by-side juxtaposition of pairs of execution graphs, supporting all of the single graph transformations, as well as new transformations to highlight graph differences and similarities.
4. Algorithms to cluster distributed executions using two approaches computed over sets of graphs: clustering by similarity to a specified graph and by number of processes.

For simplicity, we refer to any one of the libraries as XVector. ACM Transactions on Software Engineering and Methodology, Vol. 29, No. 2, Article 9. Pub. date: March 2020.

ShiViz, a robust, web-deployed, freely available implementation of the above contributions for developers to use for distributed system development. ShiViz is available online: http://bestchai.bitbucket.io/shiviz/. A video demonstrating key features of ShiViz is also available: http://bestchai.bitbucket.io/shiviz-demo/. ShiViz is being actively used in the research community and by projects within companies like Microsoft and by popular open source projects like Akka.

Four implementations of XVector, easy-to-use libraries for logging distributed system executions, for C, C++, Java, and Go. These libraries are available online: https://github.com/

The rest of this article is structured as follows: Section 2 illustrates distributed systems challenges with three use cases. Section 3 presents distributed systems background. Section 4 presents ShiViz's mechanisms for navigating and manipulating the visualization, which address the challenges of distributed system understanding. Section 5 describes our XVector and ShiViz implementations. Section 6 evaluates XVector and ShiViz with a series of controlled experiments and user studies. Section 7 discusses the threats to the validity of our study. Section 8 describes the work's limitations and future work. Section 9 places our work in the context of related research.

EXAMPLE USE CASES
This section presents three use cases that highlight how ShiViz supports developers in debugging distributed systems. These use cases were inspired by our own experiences in building distributed systems, as well as by conversations with real distributed system developers.

1. A developer attempts to understand why two leader hosts are active simultaneously in her implementation of a leader election algorithm.
A safety invariant in a leader election algorithm is that at any given time, there is at most one leader in the system. If a leader fails (such as crashing or becoming disconnected from the network), then a new leader is elected. Consider a developer whose implementation sometimes generates a situation in which two leaders are active simultaneously. The developer wants to understand under what circumstances this situation occurs.
It is standard practice for the developer to add print statements to capture critical state transitions of the system. For example, when a host becomes the leader, the host logs a message to that effect. To use ShiViz, the developer takes three additional actions: (1) The developer instruments the system using the XVector library, which augments each logged message generated by the system with a vector timestamp. (2) The developer deploys the instrumented systemcorrectly spelled and formatted.

issues in these three use cases without ShiViz, a developer today would (1) add logging code, such as print statements, automatically or manually to the system, and (2) study the resulting textual logs, one per node in the system. As a result, without ShiViz, the developer cannot easily understand the partial ordering of concurrent events in the system and has to piece together what happened by scanning through the individual node logs. Section 9 considers specific tool alternatives and prior research work in more detail. 

Summary. These three use cases illustrate the utility of ShiViz's mechanisms for (1) understanding the relative ordering of events in an execution, (2) querying for patterns of interaction between hosts in an execution, and (3) identifying structural similarities and differences between pairs of executions.

BACKGROUND: DISTRIBUTED SYSTEMS

This section overviews distributed systems concepts necessary to understand our work. A distributed system is composed of a number of hosts, and each host generates a totally ordered sequence of events. Each event is associated with a set of attributes, such as the time the event occurred. A host trace is the set of all events generated at one host.