To evaluate RQ1-RQ3, we designed a controlled experiment with 39 participants. We first designed a 15-question questionnaire (Figure 9) about four distributed executions of three systems: (1) a reliable broadcast protocol (RBcast), (2) a Facebook client-server interaction, (3) an enterprise distributed system called Voldemort [112] (Vold), and (4) a pair of Facebook interactions (FB pair). Figure 2 shows a visualization of the Voldemort log (with some hidden from view). These questions were based on our experience teaching a fourth-year distributed systems course at the University of British Columbia. In this course, the students work in teams of 24 students to build complex distributed systems in Go. As they develop their systems, the students frequently run into debugging challenges and ask questions in office hours and in an online discussion forum.

The questions in our study build on the student questions in the course, combined with our expertise as educators aimed to design questions that accurately measure system comprehension. A control group of 15 participants relied on the raw logs to answer the questions. These logs contained vector timestamps. We used raw logs as our baseline because we know of no other log analysis tool for understanding communication patterns in distributed systems. Our baseline is what a professional developer would use today [20, 40, 85]. A treatment group of 24 participants received a 10-minute introduction to ShiViz, then used the same logs and the ShiViz tool to answer the questions. None of the participants in the treatment group had prior exposure to ShiViz. Participants in both the control and treatment groups were limited to 60 minutes. The participants were assigned to the treatment and control groups at random; this randomness led to the uneven split.

The controlled study included a mix of six graduate students, eight undergraduate students, and one professor. The treatment study included 24 students in a combined graduate and undergraduate course: eight graduate students and 16 undergraduate students. Figure 10 reports the demographics of the two groups. The participants in both groups were non-experts in the field of distributed systems, though about 50% had prior experience with such systems, and about 60% had taken a related course. All participants were students or employees at UMass Amherst (and none are authors of this article).

For each group and for each question, Figure 9 reports the percentage of participants who answered the question correctly, and the Fisher's exact test p-value showing whether there is a statistically significant difference between the treatment and control groups.

Overall effect of using ShiViz: First, we compared the distribution of correctly answered questions by the treatment group to that of the control group using the two-tailed unpaired t-test. The test rejected the null hypothesis that these groups came from the same distribution (p < 0.00002). We thus conclude that using ShiViz did significantly impact the participants' ability to answer distributed systems questions correctly. The effect size of the impact is very large (Cohen's d = 1.56). We then, for each question, compared the distributions of correct answers and measured if ShiViz's effect was statistically significant for that question. We used the non-parametric two-tailed Fisher's exact test that makes no assumptions about the underlying distribution and is recommended for small sample sizes. We found that for six of the fifteen questions, the test confirmed that the treatment group and control group were statistically significantly different (p < 0.05, highlighted in Figure 9). For each of these six questions, participants using ShiViz answered more questions correctly than the participants in the control group. For these six questions, on average, 52% more of the participants using ShiViz answered questions correctly than participants in the control group.

UI: Diagram comprehension: The treatment participants scored 83%-92% (mean 90%) on questions that tested the participants' understanding of the visual components making up the execution graph as well as the relation of these components to information captured in the input log. This was an improvement over the control group, which scored 47%-100% (mean 73%). On some questions, such as question 1.1, the control group did better. We believe the reason for this is the lack of experience with ShiViz. To answer questionAnd, Fisher's exact test shows that for 6 of the 15 questions, there was a statistically significant difference (p < 0.05) in how the two groups performed. 
Developing Distributed Systems
6.2.1 Distributed Systems Course Study.
We conducted a study using an undergraduate distributed systems course in which 70 participants used ShiViz. None of the authors were directly involved in the instruction of this course. The course instructor presented vector clocks in a lecture and then briefly demonstrated how ShiViz can be used to visualize a log. The students were then given two homework assignments that described the file format ShiViz uses. One assignment asked the students to implement the bully leader election algorithm [16], and the other the distributed two-phase commit transaction algorithm [8]. As part of the assignments, the students were asked to write a short description of their experience using ShiViz within the assignments. Once the course ended, the instructor shared the student's assignments with us. More information about the two assignments is available online [13].