We then, for each question, compared the distributions of correct answers and measured if ShiViz’s effect was statistically significant for that question. We used the non-parametric two- tailed Fisher’s exact test that makes no assumptions about the underlying distribution and is rec- ommended for small sample sizes. We found that for six of the fifteen questions, the test confirmed that the treatment group and control group were statistically significantly different (p < 0.05, highlighted in Figure 9). For each of these six questions, participants using ShiViz answered more questions correctly than the participants in the control group. For example, 96% of the participants using ShiViz answered question 2.1 correctly, whereas only 33% of the participants without ShiViz did so. For these six questions, on average, 52% more of the participants using ShiViz answered

5We use the standard effect size interpretation of Cohen’s d: d ≥ .01 is interpreted as very small, d ≥ .2 as small, d ≥ .5 as medium, d ≥ .8 as large, d ≥ 1.2 as very large, and d ≥ 2.0 as huge [104].

ACM Transactions on Software Engineering and Methodology, Vol. 29, No. 2, Article 9. Pub. date: March 2020.

Visualizing Distributed System Executions

questions correctly than participants in the control group. For the other questions, the difference between the groups was not statistically significant, likely an artifact of our study’s size.

UI: Diagram comprehension. The treatment participants scored 83%–92% (mean 90%) on questions that tested the participants’ understanding of the visual components making up the execution graph as well as the relation of these components to information captured in the input log. This was an improvement over the control group, which scored 47%–100% (mean 73%). On some questions, such as question 1.1, the control group did better. We believe the reason for this is the lack of experience with ShiViz. To answer question 1.1 they had to simply count the number of host boxes, or timelines, in the diagram.

RQ1: Understanding event ordering. A primary purpose of the ShiViz execution graph is to help developers reason about logged events and inter-host interactions (Sections 4.1.1 and 4.1.2). Question 1.3 required identifying the host that initiated the protocol (i.e., generated an event that caused the other hosts to generate events). For this question participants had to reason about the ordering of events. Question 1.5 required a detailed inspection of the interactions between two particular hosts. Treatment participants had success rates of 96% and 79%, respectively. Control participants did well on the first question (100%), but did significantly worse on the second question (47%). Studying event order by studying the raw log is error-prone.

RQ2: Querying the execution graph. Several questions tested the participants’ ability to use the ShiViz search features. Questions 1.4 and 3.2 both required performing a keyword search (Section 4.2.2) for a particular event string. However, question 1.4 also involved manual inspection of the graph for highlighted (matching) events and a correlation with the hosts that generated these events. By contrast, the control group participants easily searched the log for lines containing the keyword and the corresponding host. We think this explains why it had a lower score in the treatment group (63%) as compared to the control group (87%). For question 3.2, however, both

Questions 2.1 and 2.2 could both be answered using the pre-defined request-response search (Section 4.2.1) and received respective scores of 96% and 75% in the treatment group. control group participants scored poorly on these two questions, with an accuracy of just 33% 20%, respectively. Even simple inter-host behavior, such as the request-response pattern used these two questions, is difficult to identify using raw logs.

we had not yet implemented the feature to highlight differences between two executions, two executions could be viewed side-by-side (Section 4.3.1). This differencing feature was inspired by question 4.2, which was the most difficult question in the study, with only 12.5% 7% of treatment and control participants, respectively, answering it correctly. This question the participants to reason about the divergence between two Facebook execution logs. The similarity stemmed from an extra client request in one of the executions.

Most treatment participants manually compared the executions side-by-side, looking for the first point of deviation where events do not sync up between the two executions. Their answers indi- cated that inter-host communication in the two executions eventually occurs at different points, but nearly all participants failed to properly explain the divergence in terms of high-level system behavior. One participant presented a particularly elegant answer in the form of a user-defined structured search query (Section 4.2.1). The participant’s query illustrated that one of the execu-

Semantic: Conceptual understanding. Finally, we asked conceptual questions that asked par- ticipants to understand the semantics of the communication in the system. For instance, in ques- tion 2.3, the participants were asked to describe the role of host X, which processed client requests and routed them to the appropriate data center host. Both groups did similarly on this question.

ACM Transactions on Software Engineering and Methodology, Vol. 29, No. 2, Article 9. Pub. date: March

However, for questions 3.3 and 3.4, which asked the participants to identify a host that acted similarly to another host, participants in the treatment group did significantly better (mean 61%) than those in the control group (mean 7%). The ShiViz tool helped the participants to visualize the