While the goal of our ShiViz evaluations was to evaluate a set of user interface features and underlying algorithms, such evaluations and interface design are inherently iterative. The devel- opers’ interactions with the tool inform the tool makers and suggest new interface features. For example, our studies revealed that developers might benefit from highlighting differences between two executions, as described in Section 4.3.1. We elected to implement this feature as part of ShiViz and include the description of this finding here, even though the ShiViz version used for evalua- tion did not yet include this feature: ShiViz displayed executions side-by-side but did not highlight differences. Adding such features could, in theory, alter our results, although the effect is unlikely

Our study asked participants to answer questions (Figure 9) about system behavior. The goal of the questions is to gauge the correctness of the participants’ understanding of the system be- havior. We used our expertise as educators to design questions that accurately measure system comprehension. More questions would likely have resulted in more accurate results, but the study had to balance this accuracy against the length of the study.

8 DISCUSSION

ShiViz surfaces low-level ordering information, which makes it a poor choice for understanding aggregate system behavior. The ShiViz visualization is based on logical and not real-time ordering and cannot be used to study certain performance characteristics. The ShiViz tool is implemented as a client-side-only browser application, making it portable and appropriate for analyzing sensitive

ShiViz reconstructs the happens-before relation using vector timestamps associated with logged events. The protocol for maintaining these timestamps (detailed in Section 3.2) does not provide ordering information when messages are lost. In particular, a send event without a corresponding receive event will not be identified as a send event. This is because the event that immediately happens-after the send event cannot be associated with a remote host, since the sent message was

https://github.com/p-org/Trac

isualizer.

https://github.com/akka/akka-logging-convertor.

https://github.com/tlaplus/tlaplus/issues/

ACM Transactions on Software Engineering and Methodology, Vol. 29, No. 2, Article 9. Pub. date: March 2020.



Fig. 17. A screenshot of TSViz, a tool that extends ShiViz to support real-time timestamps. A global clock ➁ time scale totally orders all events in the log. The user can interact with not only the visualized events, but also with the intervals between events.

Optimizing visualization layout. At the moment, ShiViz presents hosts in a set order: either ordering them from left to right by the number of events each host generated in total, or by their appearance in the log. We are working to formulate an algorithm that can produce a host ordering that minimizes the total number of line crossings (this problem is NP-hard).

based on ShiViz that focuses on logs with real-time timestamps in a short tool-demonstration per [83]. TSViz uses logs that are composed of both logical and real-time timestamps to give opers a better sense of when events occur relative to each other. This is in contrast to ShiViz, places each event host on a process timeline as soon as possible without violating the before requirements and using a default minimum spacing between events. Figure 17 shows screenshot of TSviz.

We envision a number of additional features to make ShiViz even more useful to developers. Real-time online visualizations. Rather than wait for an execution to complete and then collect logs from each process for visualization with ShiViz, we could augment XVector to auto- matically upload logged events to the ShiViz website as they are generated. ShiViz would then generate a scrolling visualization of the system in real-time.

Linking the visualization with code. An event in a ShiViz visualization corresponds to some program point that generated the event. A developer interested in jumping to this program point from the visualization must manually find and jump to the program point. This process could be automated and the visualization could even be integrated into IDEs.

RELATED WORK

We have briefly described ShiViz in two practitioner-oriented articles that overview the space of approaches to debugging distributed systems [14] and as a poster [1]. None of these prior publi- cations present ShiViz’s technical contributions and none evaluate ShiViz.

ACM Transactions on Software Engineering and Methodology, Vol. 29, No. 2, Article 9. Pub. date: March 2020.

There are many log analysis tools for processing totally ordered logs, including fluentd [38], logstash [76], graylog [47], splunk [111], papertrail [87], loggly [75], sumologic [113], and Zin- sight [26, 27]. A common enterprise solution for end-to-end log analysis is the ELK stack for index- ing and storing traces (elasticsearch [34]), filtering and extracting data from traces (logstash [76]), and visualization of the results (kibana [60]).

These tools can aid manual performance analysis and debugging [26], such as the detection of memory leaks in Java [30], and can be applied to the setting of streaming logs [25, 29], even at

Unlike ShiViz, the goal of these tools, including the ELK stack, excludes capturing and analyz- ing partial ordering information. As a result, unlike ShiViz, these tools do not help developers understand the exact ordering of events in a system with concurrent processes.

Visualizing Distributed Systems