Summary: Overall, we found that the participants using ShiViz had substantially higher ac- curacy than those participants who used the raw logs. Specifically, on 9 of the 15 questions those who used ShiViz had an accuracy score that was at least 19 absolute percentage points higher. And, Fisher’s exact test shows that for 6 of the 15 questions there was statistically significant difference (p < 0.05) in how the two groups performed.

Developing Distributed Systems

6.2.1 Distributed Systems Course Study. We conducted a study using an undergraduate dis- tributed systems course in which 70 participants used ShiViz. None of the authors were directly involved with the instruction of this course. The course instructor presented vector clocks in a lec- ture and then briefly demonstrated how ShiViz can be used to visualize a log. The students were then given two homework assignments that described the file format ShiViz uses. One assign- ment asked the students to implement the bully leader election algorithm [16], and the other the distributed two-phase commit transaction algorithm [8]. As part of the assignments, the students were asked to write a short description of their experience using ShiViz within the assignments. Once the course ended, the instructor shared the students’ assignments with us. More information about the two assignments is available online [13].

Leader election assignment. For the first assignment, the participants were required to im- plement a bully leader election algorithm, cause it to exhibit some interesting behavior, capture that behavior in logs, and then explain that behavior.

Many participants described how ShiViz helped them to discover, understand, and fix defects, then confirm correct behavior, in their implementation of the bully leader election algorithm. The leader election system should be robust against environmental faults such as network partitions, dropped packets, failing hosts, and slow communication links, but the initial implementations were

“The visualization helped us spot a bug that nodes were not sending their most current timestamps to other nodes until a COORD is sent.” [Group 28] (Section 4.1.1).

“One specific instance of how ShiViz helped us debug our program was when we were struggling to understand why our timestamps seemed to get increasingly misaligned over time. It was difficult to understand the ordering of our events through our Linux console, so we used ShiViz to understand the

“...we were noticing a problem where the second-highest node regularly set itself as the coordinator, even without any network failures. The highest node was also listing itself as a coordinator. We dis- covered this by examining the visualization and noticing that both the two highest nodes were often simultaneously listed as coordinators. We used this visualization to help us determine the source of

Two-phase commit assignment. For the second assignment, the participants were required to implement a system based on the two-phase commit protocol, cause their system to complete a transaction with no errors, and then cause a transaction that commits despite the failure of a

ACM Transactions on Software Engineering and Methodology, Vol. 29, No. 2, Article 9. Pub. date: March 2020.

In their reports, the participants frequently referred to the helpfulness of the visualization in confirming behavior: “these ShiViz logs show a clear indication of successful functionality of the basic 2-phase commit protocols.” [Group 18]. Some groups also noted that ShiViz helped them understand concurrency in their algorithm: “ShiViz was very useful for visualizing these parallel transactions.” [Group 15].

In both assignments, participants noted the importance of logging key events in the system, because otherwise the events are invisible to ShiViz. They also mentioned that ShiViz diagrams can be complex, particularly in cases with significant concurrency between the hosts.

6.2.2 Case Studies with Two Researchers. We performed two case studies, each one with a highly experienced systems researcher who used ShiViz to understand and debug a complex system they were working on at the time. The first researcher used ShiViz for four months while designing and debugging a distributed system for troubleshooting other systems [108]. The second researcher used ShiViz during the course of two weeks in debugging two locking implementations in the context of a commercial multi-threaded key-value store that is used as a storage engine for Mon- goDB [82]. In both cases, the researchers learned and used ShiViz on their own.

system was to minimize input sequences to distributed systems when reproducing failures; that is, his system tries to minimize external events and internal event schedules when recreating a failed execution. In building this system, the researcher used ShiViz to debug and understand behavior across a variety of distributed systems. We performed a semi-structured interview with the researcher after the four-month period to gain insight into how he used ShiViz and what features he found most useful.

The researcher initially used ShiViz to come up with a heuristic for the minimization algorithm. Instead of enumerating the entire event schedule space to find the minimal schedule, he wanted to inspect several schedules that led to the same bug and then attempt to find unifying features among these schedules. He did this by using ShiViz to visually compare (Section 4.3.1) several event schedules from three different distributed systems: a reliable broadcast implementation, a consensus protocol called Raft [86], and a distributed hash table implementation based on Pas-