Leader election assignment.
For the first assignment, the participants were required to implement a bully leader election algorithm, cause it to exhibit some interesting behavior, capture that behavior in logs, and then explain that behavior. Many participants described how ShiViz helped them to discover, understand, and fix defects, then confirm correct behavior, in their implementation of the bully leader election algorithm. The leader election system should be robust against environmental faults such as network partitions, dropped packets, failing hosts, and slow communication links, but the initial implementations were flawed. The visualization helped us spot a bug that nodes were not sending their most current timestamps to other nodes until a COORD is sent. [Group 28] (Section 4.1.1). One specific instance of how ShiViz helped us debug our program was when we were struggling to understand why our timestamps seemed to get increasingly misaligned over time. It was difficult to understand the ordering of our events through our Linux console, so we used ShiViz to understand the...we were noticing a problem where the second-highest node regularly set itself as the coordinator, even without any network failures. The highest node was also listing itself as a coordinator. We discovered this by examining the visualization and noticing that both the two highest nodes were often simultaneously listed as coordinators. We used this visualization to help us determine the source of...

Two-phase commit assignment.
For the second assignment, the participants were required to implement a system based on the two-phase commit protocol, cause their system to complete a transaction with no errors, and then cause a transaction that commits despite the failure of an ACM Transactions on Software Engineering and Methodology, Vol. 29, No. 2, Article 9. Pub. date: March 2020. In their reports, the participants frequently referred to the helpfulness of the visualization in confirming behavior: these ShiViz logs show a clear indication of successful functionality of the basic 2-phase commit protocols. [Group 18]. Some groups also noted that ShiViz helped them understand concurrency in their algorithm: ShiViz was very useful for visualizing these parallel transactions. [Group 15]. In both assignments, participants noted the importance of logging key events in the system because otherwise, the events are invisible to ShiViz. They also mentioned that ShiViz diagrams can be complex, particularly in cases with significant concurrency between the hosts.

6.2.2 Case Studies with Two Researchers.
We performed two case studies, each one with a highly experienced systems researcher who used ShiViz to understand and debug a complex system they were working on at the time. The first researcher used ShiViz for four months while designing and debugging a distributed system for troubleshooting other systems [108]. The second researcher used ShiViz during the course of two weeks in debugging two locking implementations in the context of a commercial multi-threaded key-value store that is used as a storage engine for MongoDB [82]. In both cases, the researchers learned and used ShiViz on their own.

The system was to minimize input sequences to distributed systems when reproducing failures; that is, his system tries to minimize external events and internal event schedules when recreating a failed execution. In building this system, the researcher used ShiViz to debug and understand behavior across a variety of distributed systems. We performed a semi-structured interview with the researcher after the four-month period to gain insight into how he used ShiViz and what features he found most useful.

The researcher initially used ShiViz to come up with a heuristic for the minimization algorithm. Instead of enumerating the entire event schedule space to find the minimal schedule, he wanted to inspect several schedules that led to the same bug and then attempt to find unifying features among these schedules. He did this by using ShiViz to visually compare (Section 4.3.1) several event schedules from three different distributed systemsOne thread releasing the lock and another thread acquiring that lock were non-uniform. This was contrary to the assumption that each time the acquiring thread is spinning, as threads are sent the wake signal. The problem was that a thread may take a while to wake up: it might not wake up early enough to enter the spinning state when the lock becomes available. This explained why the lock wait times were longer than expected. In addition to the acquisition time disparity, the researcher used ShiViz to determine that (1) one thread does different work than the others (based on the amount of time spent in this thread). By examining the code, the researcher realized that this thread is a work producer. (2) Using ShiViz, the researcher realized that the pthread mutex is unfair relative to the ticket-based lock; some threads acquire the lock more often than others when using the pthread mutex, which creates a priority inversion. And (3) the priority inversion makes the work producer thread run more often when using the pthread mutex than when using the ticket lock. Since the work producer is the bottleneck in the studied workload, the workload ends up completing faster with pthread.

In summary, the researcher noted that ShiViz helped her solve the problem faster than if she used other tools: "I seriously doubt that I would have been able to pinpoint the precise cause of the problem without ShiViz, and even if I had, it would have taken me many weeks."