iz to developers in work. Across these studies, we collected the developers’ impressions via surveys

We ran a case study with two systems researchers who were developing complex dis- tributed systems to evaluate the end-to-end usefulness of ShiViz to developers in their work. Across these studies, we collected the developers’ impressions via surveys and

Our evaluation results demonstrate that ShiViz supports both novice and advanced developers in distributed system development tasks. For example, our controlled experiment with 39 partici- pants showed that those using ShiViz answered statistically significantly more distributed system understanding questions correctly than control-group participants without ShiViz, with a very large effect size. The two case studies provide qualitative data about the ShiViz developer experi- ence, indicating that ShiViz helped these participants solve their problems faster than if they were to use other tools.

This article’s main research contributions are:

A new method (and the supporting open

source implementation for systems written in C, C++, Java, and Go, as detailed below) for logging and analyzing distributed systems to support common system

understanding developer tasks.  Advanced, composable graph transformations for manipulating distributed system execu

tion graphs, including constrained custom structured search and filtering by process.  A mechanism for side

by

side juxtaposition of pairs of execution graphs, supporting all of the single

graph transformations, as well as new transformations to highlight graph differ

ences and similarities.  Algorithms to cluster distributed executions using two approaches computed over sets of graphs: clustering by similarity to a specified graph and by number of processes.

1For simplicity, we refer to any one of the libraries as XVector.

ACM Transactions on Software Engineering and Methodology, Vol. 29, No. 2, Article 9. Pub. date: March 2020.

Sh

iz, a robust, web-deployed, freely available implementation of the above contributions for developers to use for distributed system development. Sh

iz is available online: http:// bestchai.bitbucket.io/shiviz/. A video demonstrating key features of Sh

iz is also available: http://bestchai.bitbucket. io/shiviz-demo/. Sh

iz is being actively used in the research community [

,

] and by projects within companies like Microsoft

and by popular open source projects like Akka.

Four implementations of XVector, easy-to-use libraries for logging distributed system exe- cutions, for C, C++, Java, and Go. These libraries are available online: https://github.com/

ShiViz is being actively used in the research community [82, 107] and by projects within companies like Microsoft2 and by popular open source projects like Akka.3 Four implementations of XVector, easy-to-use libraries for logging distributed system exe- cutions, for C, C++, Java, and Go. These libraries are available online: https://github.com/

The rest of this article is structured as follows: Section 2 illustrates distributed systems chal- lenges with three use cases. Section 3 presents distributed systems background. Section 4 presents ShiViz’s mechanisms for navigating and manipulating the visualization, which address the chal- lenges of distributed system understanding. Section 5 describes our XVector and ShiViz imple- mentations. Section 6 evaluates XVector and ShiViz with a series of controlled experiments and user studies. Section 7 discusses the threats to the validity of our study. Section 8 describes the work’s limitations and future work. Section 9 places our work in the context of related research.

EXAMPLE USE CASES

This section presents three use cases that highlight how ShiViz supports developers in debugging distributed systems. These use cases were inspired by our own experiences in building distributed systems, as well as by conversations with real distributed system developers.

1. A developer attempts to understand why two leader hosts are active simultaneously in her imple- mentation of a leader election algorithm.

A safety invariant in a leader election algorithm is that at any given time, there is at most one leader in the system. If a leader fails (such as crashing or becoming disconnected from the network), then a new leader is elected.

Consider a developer whose implementation sometimes generates a situation in which two lead- ers are active simultaneously. The developer wants to understand under what circumstances this situation occurs.

It is standard practice for the developer to add print statements to capture critical state tions of the system. For example, when a host becomes the leader, the host logs a message to effect. To use ShiViz, the developer takes three additional actions. (1) The developer the system using the XVector library, which augments each logged message generated by the tem with a vector timestamp. (2) The developer deploys the instrumented system and the problem. The system writes logs augmented with vector timestamps. (3) The developer ShiViz to visualize the logs from all of the hosts.

The developer first wants to know where in the trace the system entered the invalid two-leader state. She finds events where a host became a leader via a keyword search for isLeader=true && desc=‘became leader’ (these keywords are specific to the system logging the developer has implemented). The developer observes that for this system execution, there were four matching

Transactions on Software Engineering and Methodology, Vol. 29, No. 2, Article 9. Pub. date: March 2020.

Visualizing Distributed System Executions

results. She navigates to each of the matched messages, one at a time, and finds that two of results have the same “epoch” field value, which violates a correctness property of the