Our study asked participants to answer questions (Figure 9) about the system's behavior. The goal of the questions is to gauge the correctness of the participants' understanding of the system behavior. We used our expertise as educators to design questions that accurately measure system comprehension. More questions would likely have resulted in more accurate results, but the study had to balance this accuracy against the length of the study.

DISCUSSION
ShiViz surfaces low-level ordering information, which makes it a poor choice for understanding aggregate system behavior. The ShiViz visualization is based on logical and not real-time ordering and cannot be used to study certain performance characteristics. The ShiViz tool is implemented as a client-side-only browser application, making it portable and appropriate for analyzing sensitive data.

ShiViz reconstructs the happens-before relation using vector timestamps associated with logged events. The protocol for maintaining these timestamps (detailed in Section 3.2) does not provide ordering information when messages are lost. In particular, a send event without a corresponding receive event will not be identified as a send event. This is because the event that immediately happens-after the send event cannot be associated with a remote host since the sent message was lost.

We are working on optimizing visualization layout by formulating an algorithm that can produce a host ordering that minimizes the total number of line crossings (this problem is NP-hard). We envision several additional features to make ShiViz even more useful to developers, such as real-time online visualizations and linking the visualization with code.

RELATED WORK
There are many log analysis tools for processing totally ordered logs, including Fluentd [38], Logstash [76], Graylog [47], Splunk [111], Papertrail [87], Loggly [75], Sumologic [113], and Zinsight [26, 27]. A common enterprise solution for end-to-end log analysis is the ELK stack for indexing and storing traces (Elasticsearch [34]), filtering and extracting data from traces (Logstash [76]), and visualization of the results (Kibana [60]). These tools can aid manual performance analysis and debugging [26]. 

Visualizing Distributed Systems
The most closely related systems to ShiViz are Poet [65] and DTVS [33]. Poet is a toolchain for instrumenting and then visualizing distributed systems as time-space diagrams. DTVS visualizes concurrency regions in a trace of synchronous distributed system executions. ShiViz makes several technical advances over both Poet and DTVS, including graph transformations (e.g., filter by host), keyword and structured search, multi-execution comparison, and clustering. ShiViz is also simpler to use and deploy: ShiViz does not require a server, runs in a browser, and requires nodistributed system executions, we have also discussed related work in the field. We have highlighted the strengths and weaknesses of various tools and techniques, including those that focus on performance analysis, tracing and debugging, summarizing logs, and visualizing distributed systems in integrated development environments. 

We have also identified areas where our method could be extended or complemented by other techniques, such as using natural language processing to reconstruct distributed workflow information from existing logs or incorporating techniques for managing long executions like aggregating events or data, visualizing heatmaps, bundling edges, or animating the visualization.

Overall, our method provides a valuable tool for developers to gain insight into the behavior of their distributed systems. By automating logging and providing a visual representation of the partial ordering of events in executions, ShiViz helps developers detect issues with their systems and improve their design and performance.practices in Java-based open source software projects: A replication study in Apache Software Foundation. Empirical Software Engineering, Vol. 26, No. 1, pp. 1-34. DOI: https://doi.org/10.1007/s10664-020-09820-9.

In order to enhance distributed system executions, we have developed XVector as a collection of libraries that can add partial ordering information to execution logs, and ShiViz as a web-based tool. All of our tools, experimental designs, and data are publicly available. The source code for XVector libraries is available in ACM Transactions on Software Engineering and Methodology, Vol. 29, No. 2, Article 9 (March).

We would like to thank Jenny Abrahamson for creating the initial prototypes of XVector and ShiViz, Vaastav Anand for his help in evaluating ShiViz's scalability and assisting with the development of several XVector variants, Graham St-Laurent and Matheus Nunes for their contributions to improving the ShiViz implementation, Donald Acton and Colin Scott for their evaluation of ShiViz, and all participants in our user studies.

REFERENCES

[1] Abrahamson, J., Beschastnikh, I., Brun, Y., & Ernst, M. (2013). Shedding light on distributed system executions. In Proceedings of the International Conference on Software Engineering (ICSE) Poster Track.

[2] Acharya, M., Xie, T., Pei, J., & Xu, J. (2008). Mining API patterns as partial orders from source code: From usage scenarios to specifications. In Proceedings of the European Software Engineering Conference and ACM SIGSOFT International Symposium on Foundations of Software Engineering (ESEC/FSE).

[3] Aguilera, M. K., Mogul, J. C., Wiener, J. L., Reynolds, P., & Muthitacharoen, A. (2007). Performance debugging for distributed systems of black boxes. SIGOPS Operating Systems Review, 41(5), 74-86.

[4] Almeida, P.S., Baquero, C., & Fonte, V. (2008). Interval tree clocks. In Proceedings of the International Conference on Principles of Distributed Systems (OPODIS).