5.3 ShiViz Implementation

After processing the log, ShiViz presents the developer with a view like the one in Figure 2, which is a screenshot of ShiViz for an input log from a distributed data store system called Voldemort [112]. ShiViz is implemented as a pure client-side web application, making it trivial to deploy. To use it, a developer only needs a web browser. A developer may either paste logs and regular expressions into the web form, or they can upload a local file containing the logged execution(s) and necessary regular expression(s). ShiViz implements all of the logic in JavaScript, making it highly portable. ShiViz never sends the input system logs over the network. This makes it safe to use in a cor- porate environment where logs may contain sensitive information.

ShiViz is an open source project. A public deployment can be accessed at http://bestchai. bitbucket.io/shiviz/, and the source code is available at https://bitbucket.org/bestchai/shiviz/src/.

6 EVALUATION

We evaluated ShiViz in three ways, via a 39-participant controlled experiment, via a study with 70 students using ShiViz in a distributed systems class, and via two case studies, in which one developer per study used ShiViz while working on a complex system. We also evaluate XVec- tor’s performance overhead. We study four research questions, which are based around the design features of ShiViz highlighted in Section 4:

• RQ

: Does Sh

iz help understand the relative ordering of events? • RQ

: Does Sh

iz help query for inter-host interaction patterns? • RQ

: Does Sh

iz help identify structural similarities and differences between pairs of ecutions? •

evaluate RQ1–RQ3, we designed a controlled experiment with 39 participants. We first designed 15-question questionnaire (Figure 9) about four distributed executions of three systems: (1) an of the reliable broadcast protocol (RBcast), (2) a Facebook client-server interaction (3) an enterprise distributed system called Voldemort [112] (Vold), and (4) a pair of Facebook interactions (FB pair). Figure 2 shows a visualization of the Voldemort log (with some hidden from view).

These questions were based on our experience teaching a fourth-year distributed systems course at the University of British Columbia. In this course, the students work in teams of 2–4 students to build complex distributed systems in Go. As they develop their systems, the students frequently run

ACM Transactions on Software Engineering and Methodology, Vol. 29, No. 2, Article 9. Pub. date: March



ACM Transactions on Software Engineering and Methodology, Vol. 29, No. 2, Article 9. Pub. date: March 2020.



Fig. 10. Demographics of the participants of the control and treatment groups in the controlled user study.

into debugging challenges and ask questions in office hours and in an online discussion forum. The questions in our study build on the student questions in the course, combined with our expertise as educators, aimed to design questions that accurately measure system comprehension.

questions in our study build on the student questions in the course, combined with our expertise as educators, aimed to design questions that accurately measure system comprehension. A control group of 15 participants relied on the raw logs to answer the questions. These logs contained vector timestamps. We used raw logs as our baseline, because we know of no other log analysis tool for understanding communication patterns in distributed systems. Our baseline is

contained vector timestamps. We used raw logs as our baseline, because we know of no other log analysis tool for understanding communication patterns in distributed systems. Our baseline is what a professional developer would use today [20, 40, 85].

A treatment group of 24 participants received a 10-minute introduction to ShiViz, then used the same logs and the ShiViz tool to answer the questions. None of the participants in the treat- ment group had prior exposure to ShiViz. Participants in both the control and treatment groups were limited to 60 minutes. The participants were assigned to the treatment and control groups at random; this randomness led to the uneven split.

The controlled study included a mix of 6 graduate students, 8 undergraduate students, and 1 professor. The treatment study included 24 students in a combined graduate and undergraduate course: 8 graduate students and 16 undergraduate students. Figure 10 reports the demographics of the two groups.

The participants in both groups were non-experts in the field of distributed systems, though about 50% had prior experience with such systems and about 60% had taken a related course. All participants were students or employees at UMass Amherst (and none are authors of this article).

For each group and for each question, Figure 9 reports the percent of participants who answered the question correctly, and the Fisher’s exact test p-value showing whether there is a statistically significant difference between the treatment and control groups. Overall effect of using ShiViz. First, we compared the distribution of correctly answered questions by the treatment group to that of the control group using the two-tailed unpaired t-test. The test rejected the null hypothesis that = these groups came from the same distribution (p 0.00002). We thus conclude that using ShiViz did significantly impact the participants’ ability to answer distributed systems questions correctly. The effect size of the impact is very large (Cohen’s d = 1.56).5