author:
bibliography: draft.bib
title: Sustainable computational science:
  the ReScience initiative

# Introduction

There is a replication crisis in Science . This crisis has been highlighted in fields as diverse as medicine , psychology , the political sciences , and recently in the biomedical sciences . The reasons behind such non-replicability are as diverse as the domains in which it occurs. In medicine, factors such as *study power and bias, the number of other studies on the same question, and importantly, the ratio of true to no relationships among the all relationships probed* have been highlighted as important causes . In psychology, non-replicability has been blamed on spurious p-values (p-hacking), while in the biomedical sciences , a *lack of access to full datasets and detailed protocols for both clinical and non-clinical biomedical investigation* is seen as a critical factor. The same remarks were recently issued for chemistry . Surprisingly, the computational sciences (in the broad sense) and computer sciences (in the strict sense) are no exception despite the fact they rely on code and data rather than on experimental observations, which should make them immune to the aforementioned problems.
When Collberg and colleagues decided to measure the extent of the problem precisely, they investigated the availability of code and data as well as *the extent to which this code would actually build with reasonable effort*. The results were dramatic: of the 515 (out of 613) potentially reproducible papers targeted by the study, the authors managed to ultimately run only 102 (less than 20%). These low numbers only reflect the authors' success at running the code. They did not check for correctness of the code (i.e., does the code actually implement what is advertised in the paper), nor the reproducibility of the results (does each run lead to the same results as in the paper). One example of this problem can be found in , in which the authors tried to replicate results obtained from a computational neuroscience model. Source code was not available, neither as supplementary material to the paper nor in a public repository. When the replicators obtained the source code after contacting the corresponding author, they found that it could not be compiled and would be difficult to reuse for other purposes.
Confronted with this problem, a small but growing number of journals and publishers have reacted by adopting explicit policies for data and software. Examples can be seen in the PLOS instructions on [Materials and Software Sharing](http://journals.plos.org/plosone/s/materials-and-software-sharing) and on [Data Availability](http://journals.plos.org/plosone/s/data-availability), and in the [recent announcement](https://elifesciences.org/elife-news/inside-elife-forking-software-used-elife-papers-github) by eLife on forking (creating a linked copy of) software used in eLife papers to GitHub. Such policies help to ensure access to code and data in a well-defined format but this will not guarantee reproducibility nor correctness. At the educational and methodological levels, things have started to change with a growing literature on best practices for making computations reproducible . Related initiatives such as Software and Data Carpentry are of note since their goal is *to make scientists more productive, and their work more reliable, by teaching them basic computing skills*. Such best practices could be applied to already published research codebases as well, provided the original authors are willing to take on the challenge of re-implementing their software for the sake of better science. Unfortunately, this is unlikely since the incentives for doing such time-consuming work are low or nonexistent. Furthermore, if the original authors made mistakes in their original implementation, it seems likely that they will reproduce their mistakes in any re-implementation.

# Replication and reproduction

While recognition of the replication crisis as a problem for scientific research has increased over time, unfortunately no common terminology has emerged so far. One reason for the diverse use of terms is that each field of research has its own specific technical and social obstacles on the road to publishing results and findings that can be verified by other scientists. Here we briefly summarize the obstacles that arise from the use of computers and software in scientific research, and introduce the terminology we will use in the rest of this article. We note, however, that there is some disagreement about this particular choice of terminology even among the authors of this article.
*Reproducing* the result of a computation means running the same software on the same input data and obtaining the same results. The goal of a reproduction attempt is to verify that the computational protocol leading to the results has been recorded correctly. Performing computations reproducibly can be seen as a form of provenance tracking, the software being a detailed record of all data processing steps.

In theory, computation is a deterministic process and exact reproduction should therefore be trivial. In reality, it is very difficult to achieve because of the complexity of today's software stacks and the tediousness of recording all interactions between a scientist and a computer . explain how difficult it can be to reproduce a two-year-old computation even though all possible precautions were taken at the time to ensure reproducibility. The most frequent obstacles are the loss of parts of the software or input data, lack of a computing environment that is sufficiently similar to the one used initially, and insufficient instructions for making the software work. An obstacle specific to numerical computations is the use of floating-point arithmetic, whose rules are subject to slightly different interpretations by different compilers and runtime support systems. A large variety of research practices and support tools have been developed recently to facilitate reproducible computations. For a collection of recipes that have proven useful, see .

Publishing a reproducible computational result implies publishing all the software and all the input data, or references to previously published software and data, along with the traditional article describing the work. An obvious added value is the availability of the software and data, which helps readers to gain a better understanding of the work, and can be re-used in other research projects. In addition, reproducibly published results are more trustworthy, because many common mistakes in working with computers can be excluded: mistyping parameter values or input file names, updating the software but forgetting to mention the changes in the description of the method, planning to use one version of some software but actually using a different one, etc.

Strictly speaking, reproducibility is defined in the context of identical computational environments. However, useful scientific software is expected to be robust with respect to certain changes in this environment. A computer program that produces different results when compiled using different compilers, or run on two different computers, would be considered suspect by most practitioners, even if it were demonstrably correct in one specific environment. Ultimately it is not the software that is of interest for science, but the models and methods that it implements. The software is merely a vehicle to perform computations based on these models and methods. If results depend on hard-to-control implementation details of the software, their relation to the underlying models and methods becomes unclear and unreliable.
*Replicating* a published result means writing and then running new software based on the description of a computational model or method provided in the original publication, and obtaining results that are similar enough to be considered equivalent. What exactly "similar enough" means strongly depends on the kind of computation being performed, and can only be judged by an expert in the field. The main obstacle to replicability is an incomplete or imprecise description of the models and methods.

Replicability is a much stronger quality indicator than reproducibility. In fact, reproducibility merely guarantees that all the ingredients of a computation are well documented. It does not imply that any of them are correct and/or appropriate for implementing the models and methods that were meant to be applied, nor that the descriptions of these models and methods are correct and clear. A successful replication shows that two teams have produced independent implementations that generate equivalent results, which makes serious mistakes in either implementation unlikely. Moreover, it shows that the second team was able to understand the description provided by the first team.

Replication can be attempted for both reproducible and non-reproducible results. However, when an attempt to replicate non-reproducible work fails, yielding results too different to be considered equivalent, it can be very difficult to identify the cause of the disagreement. Reproducibility guarantees the existence of a precise and complete description of the models and methods being applied in the original work, in the form of software source code, which can be analyzed during the investigation of any discrepancies. The holy grail of computational science is therefore a reproducible replication of reproducible original work.

# The ReScience initiative

Performing a replication is a daunting task that is traditionally not well rewarded. Nevertheless, some people are willing to replicate computational research. The motivations for doing so are very diverse (see Box 1). Students may want to familiarize themselves with a specific scientific domain, and acquire relevant practical experience by replicating important published work. Senior researchers may critically need a specific piece of code for a research project and therefore re-implement a published computational method. If these people write a brand new open source implementation of already published research, it is likely that this new implementation will be of interest for other people as well, including the original authors. The question is where to publish such a replication. To the best of our knowledge, no major journal accepts replications in computational science for publication. This was the main motivation for the creation of the ReScience journal ([rescience.github.io](https://rescience.github.io)) by Konrad Hinsen and Nicolas P. Rougier in September 2015.

ReScience is an openly-peer-reviewed journal that targets computational research and encourages the explicit replication of already published research. In order to provide the largest possible benefit to the scientific community, replications are required to be reproducible and open-source. In two years of existence, 17 articles have been published and 4 are currently under review ([\#20](https://github.com/ReScience/ReScience-submission/pull/20), [\#39](https://github.com/ReScience/ReScience-submission/pull/27), [\#41](https://github.com/ReScience/ReScience-submission/pull/30), [\#43](https://github.com/ReScience/ReScience-submission/pull/30)). The editorial board covers a wide range of computational sciences (see <http://rescience.github.io/board/>) and more than 70 volunteers have registered to be reviewers. The scientific domains of published work are computational neuroscience, neuroimaging, computational ecology and computer graphics, with a majority in computational neuroscience. The most popular programming languages are Python and R. The review process takes about 100 days on average and involves about 50 comments. There is a strong bias towards successful replication (100%); experience has taught us that researchers are reluctant to publish failed replications, even when they can prove that the original work is wrong. For young researchers, there is a social/professional risk in publishing articles that show results from a senior researcher to be wrong. Until we implement a certified anonymized submission process, this strong bias will most likely remain.
One of the specificities of the ReScience journal is a publishing chain that is radically different from any other traditional scientific journal, since ReScience lives on GitHub, a platform originally designed for collaborative software development. A ReScience submission is treated very similarly to a contribution to an Open Source software project. One of the consequences is that the whole process, from submission via reviewing to publication, is open for anyone to see and even comment on.
Each submission is considered by a member of the editorial board, who may decide to reject the submission if it does not respect the formal publication criteria of ReScience. A submission must contain

-   a precise reference to the work being replicated,

-   an explanation of why the authors think they have replicated the paper (same figures, same graphics, same behavior, etc.) or why they have failed.

-   a description of any difficulties encountered during the replication,

-   open-source code that produces the replication results,

-   an explanation of this code for human readers.

A complete submission therefore consists of both computer code and an accompanying article, which are sent to ReScience in the form of a pull request (the process used on GitHub to submit a proposed modification to a software project). Partial replications that cover only some of the results in the original work are acceptable, but must be justified.
If the submission respects these criteria, the editor assigns it to two reviewers for further evaluation and tests. The reviewers evaluate the code and the accompanying material in continuous interaction with the authors through the discussion section until both reviewers consider the work acceptable for publication. The goal of the review is thus to help the authors meet the ReScience quality standards through discussion. Since ReScience targets replication of already published work, the criteria of importance or novelty applied by most traditional journals are irrelevant.

For a successful submission (i.e. partial or full replication) to be accepted, both reviewers must consider it reproducible and a valid replication of the original work. As we explained earlier, this means that the reviewers

-   are able to run the proposed implementation on their computers,

-   obtain the same results as indicated in the accompanying paper,

-   consider these results sufficiently close to the ones reported in the original paper being replicated.

For a failure to replicate submission to be accepted, we require extra steps to be taken. In addition to scrutiny of the submission by reviewers and editors, we will try to contact the authors of the original research, and issue a challenge to the community to spot and report errors in the new implementation. If no errors are found, the submission will be accepted and the original research will be declared non-replicable.

Since independent implementation is a major feature of replication work, ReScience does not allow authors to submit replications of their own research, nor the research of close collaborators. Moreover, replication work should be based exclusively on the originally published paper, although exceptions are admitted if properly documented in the replication article. Mistakes in the implementation of computational models and methods are often due to biases that authors invariably have, consciously or not. Such biases will inevitably carry over to a replication. Perhaps even more importantly, cross-fertilization is generally useful in research, and trying to replicate the work of one's peers might pave the way for a future collaboration, or may give rise to new ideas as a result of the replication effort.

# Lessons learned

Although ReScience is still a young project, the submissions handled so far already provide valuable experience concerning the reproducibility and replicability of computational work in scientific research.

## Short-term and long-term reproducibility

While some of the reasons for non-reproducibility are specific to each scientific domain, our experience has shown that there are also some common issues that can be identified. Missing code and/or data, undocumented dependencies, and inaccurate or imprecise description appear to be characteristic of much non-reproducible work. Moreover, these problems are not always easy to detect even for attentive reviewers, as we discovered when some articles published in ReScience turned out to be difficult to reproduce for someone else for exactly the reasons listed above. ReScience reviewers are scientists working in the same domain as the submitting authors, because familiarity with the field is a condition for judging if a replication is successful. But this also means that our reviewers share a significant common background with the authors, and that background often includes the software packages and programming languages adopted by their community. In particular, if both authors and reviewers have essential libraries of their community installed on their computers, they may not notice that these libraries are actually dependencies of the submitted code. While solutions to this problem evidently exist (ReScience could, for example, request that authors make their software work on a standard computational environment supplied in the form of a virtual machine), they represent an additional effort to authors and therefore discourage them from submitting replication work to ReScience. Moreover, the evaluation of *de-facto* reproducibility ("works on my machine") by reviewers is useful as well, because it tests the robustness of the code under small variations in the computational environments that are inevitable in real life. Our goal is to develop a set of recommendations for authors that represent a workable compromise between reproducibility, robustness, and implementation effort. These recommendations will evolve over time, and we hope that with improving technology we will ultimately reach full reproducibility over a few decades.
Another issue with reproducibility is that with today's computing technology, long-term reproducibility can only be achieved by imposing drastic constraints on languages and libraries that are not compatible with the requirements of research computing. This problem is nicely illustrated by whose authors report trying to reproduce their own work performed two years earlier. Even though Barba's group is committed to reproducible research practices, they did not escape the many problems one can face when trying to re-run a piece of code. As a consequence, code that is written for ReScience today will likely cease to be functional at some point in the future. The long-term value of a ReScience publication lies not just in the actual code but also in the accompanying article. The combination of the original article and the replication article provide a complete and consistent description of the original work, as evidenced by the fact that replication was possible. Even 5, 10, or 20 years later, a competent scientist should be able to replicate the work again thanks to these two articles. Of course, the new code can also help, but the true long-term value of a replication is the accompanying article.

## Open reviewing

The well-known weaknesses of the traditional anonymous peer-reviewing system used by most scientific journals have motivated many experiments with alternative reviewing processes. The variant adopted by ReScience is similar to the ones used by F1000Research or PeerJ, but is even more radically open: anyone can look at ReScience submissions and at the complete reviewing process, starting from the assignment of an editor and the invitation of reviewers. Moreover, anyone with a GitHub account can intervene by commenting. Such interventions could even be anonymous because a GitHub account is not required to advertise a real name or any other identifying element. ReScience does currently require all authors, editors, and reviewers to provide real names (which however are not verified in any way), but there are valid reasons to allow anonymity for authors and reviewers, in particular to allow junior scientists to criticize the work of senior colleagues without fear of retribution, and we envisage exploring such options in the future.

Our experience with this open reviewing system is very positive so far. The exchanges between reviewers and authors are constructive and courteous, without exception. They are more similar in style to a coffee-table discussion than to the judgement/defence style that dominates traditional anonymous reviewing. Once reviewers have been invited and have accepted the task, the editors' main role is to ensure that the review moves forward, by gently reminding everyone to reply within reasonable delays. In addition, the editors occasionally answer questions by authors and reviewers about the ReScience publishing process.

The possibility to involve participants beyond the traditional group of authors, editors, and reviewers is particularly interesting in the case of ReScience, because it can be helpful to solicit input from the authors of the original study that is being replicated. For example, in one recent case ([\#28](https://github.com/ReScience/ReScience-submission/pull/28)), a reviewer suggested asking the author of the original work for permission to re-use an image. The author intervened in the review and granted permission.

## Publishing on the GitHub platform

[GitHub](http://github.com/) is a commercial platform for collaborative software development based on the popular version control system [git](https://git-scm.com/). It offers unlimited free use to public projects, defined as projects whose contents are accessible to everyone. All ReScience activities are organized around a few such Open Source projects hosted by GitHub. This is an unusual choice for a scientific journal, the only other journal hosted on GitHub being The Journal of Open Source Software . In this section, we discuss the advantages and problems resulting from this choice, considering both technical and social issues.

There are clear differences between platforms for software development, such as GitHub, and platforms for scientific publishing, such as [HighWire](http://home.highwire.org/). The latter tend to be expensive commercial products developed for the needs of large commercial publishers, although the market is beginning to diversify with products such as [Episciences](https://www.episciences.org/). More importantly, to the best of our knowledge, no existing scientific publishing platform supports the submission and review of code, which is an essential part of every ReScience article. For this reason, the only option for ReScience was to adopt a software development platform and develop a set of procedures that make it usable for scientific publishing.

Our experience shows that the GitHub platform provides excellent support for the reviewing process, which is not surprising given that the review of a scientific article containing code is not fundamentally different from the review of code with accompanying documentation. One potential issue for other journals envisaging adoption of this platform is the necessity that submitting authors have a basic knowledge of the version control system Git and of the techniques of collaborative software development. Given the code-centric nature of ReScience, this has not been a major problem for us, and the minor issues have been resolved by our editors providing technical assistance to authors. It is of course possible that potential authors are completely discouraged from submitting to ReScience by their lack of the required technical competence, but so far nobody has provided feedback suggesting that this is a problem.

The main inconvenience of the GitHub platform is its almost complete lack of support for the publishing steps, once a submission has successfully passed the reviewing process. At this point, the submission consists of an article text in Markdown format plus a set of code and data files in a git repository. The desired archival form is an article in PDF format plus a permanent archive of the submitted code and data, with a Digital Object Identifier (DOI) providing a permanent reference. The [Zenodo](https://zenodo.org/) platform allows straightforward archiving of snapshots of a repository hosted on GitHub, and issues a DOI for the archive. This leaves the task of producing a PDF version of the article, which is currently handled by the managing editor of the submission, in order to ease the technical burden on our authors.

A minor inconvenience of the GitHub platform is its implementation of code reviews. It is designed for reviewing contributions to a collaborative project. The contributor submits new code and modifications to existing code in the form of a "pull request", which other project members can then comment on. In the course of the exchanges, the contributor can update the code and request further comments. Once everybody is satisfied, the contribution is "merged" into the main project. In the case of ReScience, the collaborative project is the whole journal, and each article submission is a contribution proposed as a pull request. This is, however, not a very intuitive representation of how a journal works. It would be more natural to have a separate repository for each article, an arrangement that would also facilitate the final publishing steps. However, GitHub does not allow code review on a new repository, only on contributions to an already existing one.

Relying on a free-use offer on a commercial platform poses some additional problems for scientific publishing. GitHub can change its conditions at any time, and could in principle delete or modify ReScience contents at any time without prior notice. Moreover, in the case of technical problems rendering ReScience contents temporarily or permanently inaccessible, the ReScience community has no legal claims for compensation because there is no contract that would imply any obligations for GitHub. It would clearly be imprudent to count on GitHub for long-term preservation of ReScience content, which is why we deposit accepted articles on Zenodo, a platform designed for archiving scientific information and funded by research organizations as an element of public research infrastructure.

The use of free services provided by GitHub and Zenodo was clearly important to get ReScience started. The incentives for the publication of replication work being low, and its importance being recognized only slowly in the scientific community, funding ReScience through either author page charges or grants would have created further obstacles to its success. A less obvious advantage of not having to organize funding is that ReScience can exist without being backed by any legal entity that would manage its budget. This makes it possible to maintain a community spirit focused on shared scientific objectives, with nobody in a position to influence ReScience by explicit or implicit threats of reducing future funding.

# Outlook

Based on our experience with the ReScience initiative, we can engage in informed speculation about possible future evolutions in scientific publishing, in particular concerning replication work. We will not discuss minor technical advances such as a better toolchain for producing PDF articles, but concentrate on long-term improvements in the technology of electronic publishing and, most of all, in the attitude of the scientific community towards the publication, preservation, and verification of computer-aided research.

A fundamental technical issue is the difficulty of archiving or accurately describing the software environments in which computational scientists perform their work. A publication should be accompanied by both a human-readable description of this environment and an executable binary form. The human-readable description allows an inspection of the versions of all software packages that were used, for example to check for the impact of bugs that become known only after a study was published. The executable version enables other scientists to re-run the analyses and inspect intermediate results. Ideally, the human-readable description would permit rebuilding the executable version, in the same way that software source code permits rebuilding executable binaries. This approach is pursued for example by the package manager Guix . A more limited but still useful implementation of the same idea exists in the form of the conda package manager , which uses a so-called environment file to describe and reconstruct environments. The main limitation compared to Guix is that the packages that make up a conda environment are themselves not reproducible. For example, a conda environment file does not state which compiler versions were used to build a package.

Containerization, as implemented e.g. by Docker is currently much discussed, but provides only the executable version without a human-readable description. Moreover, the long-term stability of the container file format remains to be evaluated. History has shown that long-term stability in computing technology is achieved only by technology for which it is a design priority, as in the case of the Java Virtual Machine . Docker, on the contrary, is promoted as a deployment technology with no visible ambition towards archiving of computational environments.

Today's electronic publishing platforms for scientific research still show their origins in paper-based publishing. Except for the replacement of printed paper by a printable PDF file, not much has changed. Although it is increasingly realized that software and data should be integral parts of most scientific publications today, they are at best relegated to the status of "supplementary material", and systematically excluded from the peer review process. In fact, to the best of our knowledge, ReScience is the only scientific journal that aims to verify the correctness of scientific software. As our experience has shown, it is far easier to graft publication onto a software development platform than to integrate software reviewing into a publishing platform. Furthermore, tools that will allow for the automated validation of computational models and the automated verification of correctness are being actively developed in the community (see, for example, [SciUnit](https://github.com/scidash/sciunit) or [OSB-model-validation](https://github.com/OpenSourceBrain/osb-model-validation)). An integration of such frameworks, which would greatly enhance the verification and validation process, seems feasible for the existing software development platforms.

A logical next step is to fully embrace the technology designed for software development, which far better takes into account the specificity of electronic information processing than today's scientific publishing systems. In addition to the proper handling of code, such an approach offers further advantages. Perhaps the most important one is a shift of focus from the paper as a mostly isolated and finished piece of work to scientific progress as a collection of incremental and highly interdependent steps. The [Software Heritage](https://www.softwareheritage.org/) project, whose aim is to create a permanent public archive of all publicly available software source code, adopts exactly this point of view for the preservation of software. As our experience with ReScience has shown, integrating the narrative of a scientific article into a framework designed for software development is not difficult at all. Publishing and archiving scientific research in Software Heritage would offer several advantages. The intrinsic identifiers that provide access to the contents of the archive permit unambiguous and permanent references to ongoing projects as well as to snapshots at a specific time, and to whole projects as well as to the individual files that are part of them. Such references hold the promise for better reuse of scientific information, for better reproducibility of computations, and for fairer attribution of credit to scientists who contribute to research infrastructure.

One immediate and legitimate question is to wonder to what extent a replication could be performed *prior* to the publication of the original article. This would strongly reinforce a claim because a successful and independent replication would be available right from the start. As illustrated in Figure , this would require group A to contact group B and send them a draft of their original work (the one that would be normally submitted to a journal) such that group B could perform a replication and confirm or refute the results. In case of confirmation, a certified article could be later published with both groups as authors (each group being identified according to their respective roles). However, if the replication fails and the original work cannot be fixed, this would prevent publication. This model would improve the quality of computational research and also considerably slow down the rapid pace of publication we are observing today. Unfortunately, such a scenario seems highly improbable today. The pressure to publish is so strong and the incentive for doing replication so low that it would most probably prevent such collaborative work. However, we hope that the current replication crisis will lead to a change in attitude, with an emphasis on the quality rather than the quantity of scientific ouput, with CoScience becoming the gold-standard approach to quality assurance.


author: [^1];  ;  ;  ;  ;  ;  ;  ;  ;  ;
subtitle: Columnar Object Framework For Effective Analysis
title: Coffea

# Introduction

The present challenge for High-Energy Particle Physics (HEP) data analysts is daunting: due to the success of the Large Hadron Collider (LHC) data collection campaign over Run 2 (2015-2018), the Compact Muon Solenoid (CMS) detector has amassed a dataset of order 10 billion proton-proton collision events. The raw detector information is reconstructed into high-level information, such as the trajectories of visible outgoing subatomic particles, using a centrally-maintained software  and distributed computing system . Even after significant processing and distillation, this high-level summary of each collision event still contains order 1 kilobyte of compressed data . The CMS physicist/data-analyst is tasked with processing the resulting tens of terabytes of distilled data (along with a similar magnitude of simulation data) in a mostly autonomous fashion, typically designing (or inheriting) a processing framework written in C++ or Python using a set of libraries known as the ROOT framework , and parallelizing the processing over distributed computing resources using HTCondor  or similar high-throughput computing systems. Each physicist is interested in a different subset of this data, and will look at a different projection of the high-level variables in order to produce summary statistics, such as histograms, from which statistical inference can be made to measure parameters of, or probe compatibility with, theoretical expectations.

As the collected and simulated datasets grow, the development and maintenance burden of this analysis code represents an increasingly large fraction of physicists' time. The scale of processing requires physicists to be increasingly cognizant of the structure and performance characteristics of their code, despite the fact that they are not traditionally trained as software engineers. Simultaneously, physicists are more often utilizing modern machine learning techniques and their associated libraries (e.g., Tensorflow, PyTorch), which enforce certain data structures that are atypical in traditional approaches to analysis. Several hundred physicists face these challenges now, and they will only magnify as the High-Luminosity LHC upgrade is projected to produce datasets over an order of magnitude larger. Now is the time to investigate novel approaches to HEP data analysis that are easy to use, scalable, and fast.

HEP physicists have been called upon to produce bespoke data analysis applications for several decades, historically at the forefront of data volume and computational requirements. However, the needs of the private sector, and also of other scientific disciplines, have reached or surpassed the scale of HEP data analysis. One of our core goals is to investigate the applicability of solutions found outside HEP towards our data analysis needs. In these proceedings, we introduce the concept of columnar analysis and the coffea framework, then discuss the user experience and scalability characteristics of the framework, and propose future directions for analysis systems research and development that we will pursue.

# Columnar Analysis

Columnar analysis is a paradigm that describes the way the user writes the analysis application that is best described in contrast to the the traditional paradigm in HEP of using an event loop. In an event loop, the analysis application operates in a record-oriented manner on the input data, where each reconstructed particle collision event record is deserialized and loaded into a structure containing several fields, such as the properties of the visible outgoing particles that were reconstructed in a collision event. The analysis code manipulates this structure to either output derived quantities or summary statistics in the form of histograms. In contrast, columnar analysis operates in a column-oriented manner, where individual columns represent portions of data spanning a *chunk* (i.e., partition, batch) of event records are manipulated using array programming primitives in turn, to compute derived quantities and summary statistics. A visualization of the two contrasting approaches is shown in Fig. .

Array programming is widely used within the scientific python ecosystem, supported at the foundational level by the NumPy  library, with functionality extended by libraries such as SciPy , Pandas , and others. However, although the existing scientific python stack is fully capable of analyzing rectangular arrays (i.e., no variable-length array dimensions), HEP data is very irregular, and manipulating it can become awkward without first generalizing array structure a bit. The awkward-array  library performs this function, extending array programming capabilities to the complexity of HEP data. In particular, awkward-array extends NumPy *broadcasting* semantics to so-called *jagged* arrays, where the number of elements varies per sub-array. In addition, awkward-array allows Structure-of-Array (SoA) objects to be manipulated and viewed as though they were plain structures, so that intuitive expressions can be written that access individual arrays of data.

# The coffea framework

The coffea framework is a python package that is indexed and installable [^2] via standard python packaging infrastructure, which provides several recipes and utilities to aid in development of a HEP analysis within the columnar analysis paradigm. The package has developed organically to support our initial goal of implementing CMS data analyses within the scientific python ecosystem. Here, we define "analysis" as the process of selecting events of interest based on the high-level input variables or functions thereof, extracted from ROOT files or similar column-oriented serialization formats, transforming numeric quantities, and computing summary statistics or low-volume tabulated data for use with statistical inference tools. As this is the first full attempt at using scientific python libraries for CMS analyses, certain missing extensions which normally would be provided by ROOT libraries or by CMS experiment-specific libraries had to be ported to a columnar paradigm. Examples include:

-   multidimensional histogram objects that are serializable and mergeable, with support for both categorical and numeric axes, that are fillable by arrays;

-   certain experiment-specific corrections that are applied to simulated data, typically as a piecewise function of some set of event parameters;

-   utilities to reduce boilerplate necessary to construct the awkward arrays that represent the events SoA object; and

-   wrappers to enable use of novel scale-out mechanisms, as discussed further in Section .

As many of these extensions are performance-critical, we make full use of the vectorized array programming primitives provided by NumPy and SciPy. We also utilize Numba , which just-in-time compiles a subset of python and NumPy code into machine code, for operations where no efficient composition of existing array programming primitives could be found. In general, any operation performed along the critical dimension—namely, per event—is not performed sequentially within the python interpreter but by vectorized machine code on the arrays forming the high-level SoA objects.

# User experience

All coffea core developers actively participate in CMS data analyses. Along with the core developers, several other data analysts have started adopting the coffea framework within their analysis workflows. Some users have contributed to the package, extending coffea where their analysis required additional functionality. As the package is open-source and available on GitHub [^3], any interested party can submit patches to the source to add features or contribute bugfixes.

The main feedback from end users is that they appreciate being able to write analysis code in python while still maintaining good performance, such that their workflows complete quickly. The user time and barrier to entry for python is sufficiently low for physicist-programmers that having the entire workflow code in python is a useful feature. In existing python bindings for ROOT, the lack of a vectorized infrastructure along the event dimension leads to poor-performing workflows that inhibited the ability for users to develop and test their analysis.[^4] Users who do not already have extensive experience with the ROOT ecosystem have reported that the scientific python ecosystem has excellent "google-ability", i.e., example solutions for many tasks are easily found and supporting documentation is quite extensive. By depending on the large user and developer base of scientific python, the coffea framework requires little developer effort to maintain user support, since the underlying infrastructure often provides the necessary support.

In some cases, no efficient composition of array programming primitives can provide the end user with the necessary solution for a given task. This happens when the algorithm to be implemented, or intra-event data on which it operates, becomes sufficiently complex. In these cases, users must learn how to write their own array programming kernels, e.g., with Numba. Presently, these kernels can be challenging to write. Work is ongoing within the awkward-array library to simplify such developments. For some widely-used algorithms, user-developers have contributed the missing algorithms to the coffea codebase.

One important benefit of utilizing the scientific python ecosystem is the seamless integration with Jupyter notebooks . Jupyter notebooks combine documentation, source code, and partial results into one persistent document that provides much faster iterative data exploration than the traditional command-line interface. Notebooks can be effective as teaching aids, and may enable analysis preservation efforts. However, users still find the traditional approach valuable for established workflows.

# Scalability

One of the challenges for HEP data analysts is *scale-out*: the process of scaling their workflow from a small test sample to tens of terabytes. Luckily, essentially all HEP workflows are trivially parallel as each event is statistically independent. Analysts build their own map-reduce algorithm to divide some set of input files into a set of batch jobs, and reduce the job output data by adding bin counts in histograms. With coffea, we provide a high-level wrapper around user analysis code: the coffea *processor*. The `ProcessorABC` abstract class defines an interface where the user receives a chunk of columns and is responsible for returning a reducible object, where reducibility is enforced by the `AccumulatorABC` abstract class. The most common reducible object is a histogram.

With the processor defined, the responsibility of dividing the work over a given execution resource and accumulating the results is taken by the coffea framework. The currently supported coffea executors are:

1.  a local executor utilizing parallel python processes;

2.  an execution wrapper for the Apache Spark  distributed computing platform;

3.  an execution wrapper for the Dask  distributed computing platform; and

4.  an execution wrapper for the Parsl  distributed computing platform.

The user can then run their processor on a small set of test input files or on a full dataset by simply changing the executor method within the coffea framework.

# Future directions

Our experience with the coffea executors highlights the need for intermediate-scale resources: a $\lesssim 100$ core-hour resource is sufficient to process an entire CMS analysis over run 2 data in minutes with the appropriate executor. Further speed-up will only happen with the introduction of derived column caches in a multi-user context. Therefore, we are now investigating the concept of analysis facilities, dubbed *coffea farms*, where several users operate on the same datasets, and the framework ensures that users accessing the same data will access a cached copy of it. It is envisioned that several small clusters will host coffea farm applications for small groups of CMS data analysts, enabling rapid analysis prototyping with large-scale datasets, and removing manual work to reduce the input datasets for efficient re-analysis (often referred to as *slimming and skimming*).

# Conclusions

Columnar analysis is an effective paradigm for HEP data analysis within the CMS collaboration, and we have implemented several maturing analyses in a columnar fashion. We have profited from the scientific python ecosystem to provide a library that is largely feature-complete for our needs with minimal direct developer resources. The coffea framework enables users to write high level operations, while the library code maintains performance. Coffea simplifies the interface to novel scale-out mechanisms, and provides an avenue to investigate the resource gains that may be found with a multi-user common-framework analysis facility.

[^1]:

[^2]: `pip install coffea`, other options documented at <https://coffeateam.github.io/coffea/installation.html>

[^3]: <https://github.com/CoffeaTeam/coffea>

[^4]: ROOT has also addressed this issue with the introduction of `RDataFrame`, which can help lower more analysis operations from python interpreted code into compiled machine code.


abstract: Creating an artifact in front of public offers an opportunity to involve spectators in the creation process. For example, in a live music concert, audience members can clap, stomp and sing with the musicians to be part of the music piece. Live creation can facilitate collaboration with the spectators. The questions I set out to answer are what does it mean to have liveness in interactive systems to support large-scale hybrid events that involve audience participation. The notion of liveness is subtle in human-computer interaction. In this paper, I revisit the notion of liveness and provide definitions of both *live* and *liveness* from the perspective of designing interactive systems.[^1] In addition, I discuss why liveness matters in facilitating hybrid events and suggest future research works.
author: **Sang Won Lee**

bibliography: sample.bib
title: Liveness in Interactive Systems

# Introduction

Creating an artifact — such as writing a book, developing software, or performing a piece of music — is often limited to those with domain expertise. As a consequence, effectively involving non-expert end users in such creative processes is challenging. One potential solution to this is to create an artifact *live* and to involve spectators in the creation process guided by the creators. In live creation, creators reveal the process of creation to spectators and the immediate and continuous visibility — perceptibility, to be more precise — of the live creation process helps non-experts have better understanding on the process (and the artifact) and be able to participate in the process. While increasing number of HCI literature use the term "live" interaction and "liveness", the notion of liveness is yet subtle in designing and evaluating interactive systems. I have explored ***liveness*** in various interactive systems to involve non-expert users for applications such as programming, writing, music performance, and UI design . Through the previous works that I explored, I suggest technical definitions of both "live" and "liveness' that can be used in designing and evaluation interactive systems. Also I discuss the challenges of involving users in live creation and how technologies can address such challenges.

# Definition of *Live*

In the Oxford Dictionary, the adjective *live*, in this context, means "relating to a musical performance given in concert, not on a recording" or "transmitted at the time of occurrence, not from a recording". In the Merriam-Webster Dictionary (online), *live* is defined as "of or involving a presentation (such as a play or concert) in which both the performers and an audience are physically present" or "broadcast directly at the time of production". Two common components of the above definitions are the concurrence of action and perception, and the notion of an artifact that is produced, performed, or transmitted, such as a musical performance, TV show, or sporting event. The concurrence of action and perception implies live settings include two different groups: creators — those who are in action (performers, actors, broadcasters) — and users (consumers more broadly) — those who perceive the action (an audience, viewers, listeners). In summary, there exist four components that constitutes being live: 1) an artifact, 2) those who create/deliver the artifact, 3) those who perceive it, and 4) the concurrence between 2 and 3.

I define the term ***live*** as follows:

First, the term "perceptible" is used in its broadest sense to include any capability of being perceived through any human sensory system (e.g., visible, audible, tangible, smellable, tastable). For example, a live concert can be heard over the radio. Frequently, the term live relates to the perception of multisensory information, typically audiovisual but can extend to other senses (taste, smell, touch). The process being perceptible can be naturally accomplished when creators and spectators are co-located at the same time. In this case of live creation, the co-locatedness includes the spectators have access to — can perceive — the process of creation. However, it does not necessarily mean that live creation needs to happen only in co-located setup. Creators and spectators can be remote and the tele-communication technology that is used to make the process perceptible will be required. Interestingly, the notion of liveness has developed with the advancement of media technologies through which one can transmit audiovisual information to a remote location (radio, "tele"phone, "tele"vision, and recording) For example, before recording technology, there was no way to listen to music other than live performance; all music was live music back then.

Second, being perceptible in "(near) real time" means that the process is perceptible at the time of occurrence or with minimal delay. There is no transmission delay involved for the live process for which spectators and performers are co-located is physically perceptible in real-time. Therefore, the term "(near) real time" typically relates to the latency involved in technologies that enable the perceptibility of the process and artifacts to remote spectators. The notion of (near) real time is defined as the timeliness of data in the context of distributed systems and can depend on the situation, implying that there are no significant delays . For example, having a long latency is even desirable in certain context; in typical live TV shows, a broadcast delay is intentionally added to allow undesirable content, such as profanity or nudity, to be censored . On the other hand,in the case of an interactive system (two-way communication between creators and spectators) beyond media delivery, latency can negatively impact the user experience. For example, a teleconferencing system (such as Skype) allows creators to present the process of creation live to users, but a delay of more than a few seconds would make two-way communication difficult .

# *Live* Creation

Typically creation processes are not live — in fact, they are often *hidden* to users (or consumers more broadly) entirely. The creation of an artifact takes place asynchronously with respect to its consumption, and the process of creation is separated from the users. For example, readers (users) read a book (an artifact) only after the author (its creator) has finished writing it. The same is typically true of food that people eat at a restaurant, a painting exhibited at a museum, or software written by developers. On the contrary, in live settings, the process of creating artifacts is revealed to users to an extent that is typically understandable and presentable to spectators. For example, at a live music concert, music is the *artifact*, musicians are the *creators*, and audience members are the *users*. While many aspects of a music performance are still asynchronously done or hidden — such as practices, rehearsals, back-stage efforts — the audience get to see the actual performance and perceive it as a live creation of an artifact. Such asynchronous efforts are more necessary for especially making the process live.

Many creators choose to reveal the process of creation to users so that users can understand how an artifact is made and appreciate the effort that goes into creating an artifact. In general, people put more value on an artifact that is created live. For instance, people are willing to pay more for live music performances, even though listeners are less and less willing to pay for recorded music . In the case of cooking, in some restaurants, chefs cook in front of their customers in real time. This is true of teppan-yaki restaurants (sometimes referred to as "Japanese steakhouses" in the US), which serve a style of Japanese cuisine cooked on an iron hot plate called a *teppan* . The visible process of creation adds value to the artifact because users can understand how the artifact is created .

Live creation encourages creators to reinforce the performative aspect of the creation process. Creators would want to augment their actions (i.e., narration) and to add even unnecessary steps or modifications (i.e., exaggerated movements of teppan-yaki chef) for make the process more transparent and engaging. The perceptibility on the artifact to spectators and the existence of them allows creators to express their thoughts and emotion on the artifact and to convey their creative practice. Increasing such demonstrative components of a live interaction gives creators opportunities to engage people through understanding and theatrical elements. For users standpoint, as perceiving the process takes time, the artifact now is something that can be not only used but also experienced. Creators would "perform" the creation activity live like performing arts compared to the case of non-live creation with no audience.

# Definition of *Liveness*

How, then, can we define ***liveness***? According to the Oxford Dictionary liveness is "the quality or condition (of an event, performance, etc.) of being heard, watched, or broadcast at the time of occurrence." In general, liveness has been valued, especially in the context of performing arts such as music, theatre, and dance. Auslander explored the value of liveness, particularly in the era of culture dominated by (typically non-live) mass media and media technology, such as television . Liveness also has been expanded to broader areas digital arts and new media. Crisell also states that a broadcast conveying messages over distances without the time lapse is the basis of the broadcast's liveness .

A non-live process can still incorporate liveness through the use of techniques that give the audience *the sense of being there* and the opportunity to witness what happens. A typical example of a non-live creative process that still has liveness is replay of recorded live events. Based on the definition of live used in the previous section, recording of a live music performance is NOT live due to the delay between creation and consumption. However, it was live at the time of recording, and the recording effectively shows the characteristics of live settings: a number of characteristics unique to live settings can be found, such as any risks involved, the impromptu nature of the performance, the sound of the audience cheering, and the unedited visuals, especially compared to non-live forms of them — studio-recorded music. Hook et al. describe liveness as "the properties of intimacy and immediacy experienced by both spectators and performers", which comes from the spectators' proximity to and even inclusion in a performance of some kind . The authors assert that the uses of technology can bring a sense of presence and involvement in events, which can exist independently of time and space. For example, having multiple views of a live music concert on a television is impossible to replicate for an audience member at a concert hall. Therefore, liveness can be related to the qualities and properties that help users experience the process of creating an artifact as if they are there.

I define the term ***liveness*** as follows:

This particular definition is selected in consideration of the context of interactive systems design and can be used to quantitatively assess liveness of an interactive system. Liveness can be considered in terms of three values: *immediacy*, *continuity*, and *perceptibility*. In the following subsections, I will discuss each quality of liveness.

## **Immediacy: minimizing the latency**

The most straightforward quality of liveness is immediacy. The immediacy can be simply measured by the average latency between the time of creation and the time that the process is perceptible to spectators. For example, live broadcasting of a music concert has more liveness than replaying a recording of the same concert(See Figure--2), as the former becomes perceptible sooner after the time of creation.

The value added by revealing the process of creation immediately can vary depending on the artifact that is created. In general, the artifact of which its value increases with recency — minimizing the delay between creation and consumption — can have potential increase in its value with live creation. In some cases, the recency itself can have a value in its quality — e.g., freshly cooked vs. microwaved food. The immediacy requirement can result from a highly personalized nature of artifacts usage, services, or content — e.g., mass-produced ready-made products(ready-made clothes) vs. made-to-order products(order-made dress). In these case, being able to defer the creation until the time to get feedback from the consumers is important values that therefore immediacy brings The immediacy is an essential quality that enable real-time interaction between spectators and creators as discussed in a previous section. Lastly, the value of immediacy can also come from the value of uncertainty involved in the live process — e.g., live broadcast of sports events vs. recording of them given the final result is known — and the creators' efforts to reconcile such risks — e.g., live music performance vs. videotaped performance in a studio which could have done in multiple takes.

## **Continuity in revealing the creation process**

The continuity (or continuous perceptibility) is how continuously the state of the process is perceptible to spectators. Continuity can account for why recordings of live events can still have liveness. If there is an artifact whose state is `A`, the state visible to spectators `A'` can be updated discretely or intermittently. The discrepancy between `A` and `A'` may arise not only from the latency, but also from the discontinuous visibility (or state synchronization) coming from the mediating technology (See Figure--3). Continuity can be orthogonal to immediacy of an interactive system or live events. For example, a live TV sporting event with commercial breaks is not continuously visible to viewers, because viewers cannot see what happens during the breaks, unlike the audience attending the event in person. Text editors are another software example. Suppose one wants to write text live to remote viewers, for educational purposes in a shared editor (such as Google Docs). In this shared editor, the document is continuously shared — the state is synchronized with every keystroke — even though there can be some latency. However, in the case of an instant chat messenger, messages are shared only when a user presses the Return key (or presses a send button); until that point, the typed text is not visible to the other party. In this case, the artifact (conversation) is shared discontinuously. Therefore, Google Docs has more liveness than an instant chat messenger. The notion of continuity is particularly effective to understand the liveness when certain media or processes are not live and do not occur in real time — a recording of a live event, for instance.

As an extreme example, a film can be seen as a medium that has less liveness compared to a play in a theatre. Based on the definition used here, in terms of immediacy, the final artifact may be presented to viewers with months and years of latency. Another aspect that makes film have less liveness is the discontinuous ways in which the artifact is created and presented. The total time it takes to produce a film may span years. However, only a tiny portion of the entire creative process is visible to the audience, as the typical running time of a film is from two to three hours. However, it takes one hour to perform an hour-long play, and it is visible for an hour to the audience. While a tremendous amount of pre-production process may be required (writing, scripting, rehearsal) for a theatrical performance, the fact that the one-hour process is continuously visible to the audience in real time since the beginning constitutes liveness of a theatrical performance and brings with it the impromptu and risky nature of live performance. Similar kinds of liveness coming from continuity can exists in asynchronous filming. For example, a long take in a film can increase the liveness for a particular scene.

## **Perceptibility: the sense of being there**

Lastly, perceptibility is an important characteristic of liveness. It is related to creating *the sense of being there*, typically when it is not, thus the spectators can see, hear, and feel the creative process. Therefore being co-located in real-time does create a natural perceptibility. However, the sense of being there can be augmented with technology.

In addition, the notion of perceptibility can dynamically change over time with the emergence of new media. Virtual reality can be used to create audiovisual perceptibility and many researchers are working on supplementing the other senses as well . Revisiting the example of film, a film may bring more liveness by augmenting the sense of being there, not only through well-made content, but also by media technologies, such as a surround screen, 3D cinema , spatialized audio , and haptic feedback . The technological space of enhancing perceptibility is dynamically expanding and challenging to define. These two temporal dimensions — immediacy and continuity — provide metrics for us to evaluate and compare interactive systems in terms of liveness. At the moment, there is no clear way to measure perceptibility other than asking the participants the perceived value, which can vary across individuals.

One underlying requirement for the perceptibility is that spectators should be able to understand the creation process from their perception. While the perceptibility does not include the extent to which the process of creation is easy to understand for the spectators, the effect of liveness will be minimal if the spectators cannot comprehend the process of creation. Research in live electronic music, in which performers often sit behinds a laptop computer and an audience is not aware of what is going on, highlights the challenge of delivering liveness that is decoupled from a performer's physical actions .

# Liveness for Hybrid Events

Hybrid events do not necessarily need technical components to realize. However, having computational supports can enable novel audience participation modes that were not available before. For example, I have developed two interactive systems for audience participation in music concerts . Typically, audience participation in music concerts was limited to the level in which the audience generated sound only accompanies the primary sound coming from the stage. In addition, audience participation is limited to musicians whose music is well known to their audience. On the contrary, the systems that I developed enable immediate participation and the only sound of a music piece is coming from the audience, not from the on-stage musician(s). In these settings, liveness is an important virtue for their participation as having liveness allow the participants to eventually understand how their participation contributes to creating an artifact. While the co-located nature of such event naturally brings liveness, using interactive systems to facilitate co-located collaboration sometimes add ruin to the liveness by hiding the process of creation with information lost in digitized participation. However, having liveness can be costly as it requires significant engineering efforts for multi-user software: synchronizing the state in the finest resolution for the continuous perceptibly, minimizing the latency, and being able to perceive creators actions entirely. Therefore, I suggest two types of research in understanding the effects of liveness in facilitating hybrid events.

First, we need to be able to measure the liveness of interactive systems to some extent to be able to compare interactive systems in terms of liveness. The three perspectives of liveness used in this paper can be one way to measure liveness in the interactive systems . The two temporal dimensions — immediacy and continuity — are straightforward to measure. The perceptibility — the sense of being there — can be tricky to quantify even thought it would be somewhat subjective based on the perception of spectators who experience events. In addition, how each dimension contributes to the overall liveness can be a challenging research work. Second, I wish to validate the benefits of having liveness in interactive systems for facilitating hybrid events and remote collaboration in general. The first goal is the prerequisite of the second goal. Once, liveness of interactive systems can be quantified, we will be able to assess the effects of having liveness conduct simple A/B testing by varying the level of liveness in the interactive systems that we use. I wish this paper can be a starting point to initiate the discussion of the technical definition of "liveness" in HCI.

[^1]: The large portion of this paper appear in .


abstract: In regions that practice common law, relevant historical cases are essential references for sentencing. To help legal practitioners find previous judgement easier, this paper aims to label each court judgement by some tags. These tags are legally important to summarize the judgement and can guide the user to similar judgements. We introduce a heuristic system to solve the problem, which starts from Aspect-driven Topic Modeling and uses Dependency Parsing and Constituency Parsing for phrase generation. We also construct a legal term tree for Hong Kong and implemented a sentence simplification module to support the system. Finally, we propose a similar document recommendation algorithm based on the generated tags. It enables users to find similar documents based on a few selected aspects rather than the whole passage. Experiment results show that this system is the best approach for this specific task. It is better than simple term extraction method in terms of summarizing the document, and the recommendation algorithm is more effective than full-text comparison approaches. We believe that the system has huge potential in law as well as in other areas.
author: Yuchen Liu
title: Court Judgements Labeling Using Topic Modeling and Syntactic Parsing

# Introduction

## Problem Statement

In regions that practice common law, the tribunal relies highly on previous court decisions to resolve disputes. Legal practitioners frequently need to find relevant legal judgements at work. Indexing each court judgements by labels can considerably increase the speed of court judgement recommendation, and can help users find other judgements with similar topics faster.

This paper will focus on labeling every court judgement by some tags. There are existing paid websites that provide such tags, for example Westlaw Asia. However, they maintained the tags by recruiting legal experts to label manually, which is expensive and time-consuming. Plus, users cannot select the tags to find similar documents. A better solution must be developing an NLP system that can automatically extract legally important tags from a court judgement. It leads to the question that how to build such a system and how to find similar documents using these tags.

## Literature Review

Previous work of Wu, et al. focuses on semantic search and summarization using **Topic Modeling**. It proved that the Topic Modeling based on labeled aspects (background, injury, losses, and compensations for Personal Injury cases) shows the best summarization ability. Figure shows the general process of Aspect-driven Topic Modeling.

For each aspect of documents in the training set, a few significant topics will be generated. Based on the resultant topics, the similarity between paragraphs and aspects can be calculated using the normalized inner product of the word2vec embeddings of the paragraphs and topics . This way, paragraphs that are related to each aspect can be found. By simply changing paragraphs to sentences in this algorithm, we can find the most relevant sentences to each aspect. These aspect sentences should comprehensively summarize the court judgement and will serve as the starting point of this project.

Two of the classical Syntactic Parsing approaches will be used in our model.

**Dependency Parsing** finds the asymmetric dependency relations between words. These relations can be represented as arrows pointing from the heads to the dependents. Usually, the heads are more grammatically important than dependents. For example, adjectives normally depend on nouns. The result should form a non-recurrent tree. We can backtrack from one leaf of the tree and find all its parent words.

**Constituency Parsing** uses phrase structure grammar to organize words into nested constituents. Each constituent is labeled by one of the predetermined grammar units, called POS Tags, for example, VB denotes verb phrase and PP denotes prepositional phrase.

## Objective

The tags given to each judgement should fulfill the below properties.

1.  The tags should be/contain legally important terms.

2.  The tags are free of grammatical/content mistakes that will affect understanding.

3.  The tags can navigate the user to other similar documents.

4.  The tags can describe important aspects of the current judgement.

As illustrated in 1.3, if we generate tags based on the aspect sentences, requirement 4 will be satisfied . Therefore, we will only focus on solving requirements 1-3 in the following methodology. Because of the regional difference of constitution, we will use Hong Kong as the example and use the [HKLII](https://www.hklii.hk/eng/) website as the data source for historical judgements.

## Report Outline

Chapter 2 will introduce two supporting modules and the main algorithm. The similar document recommendation algorithm will then be illustrated. Chapter 3 will show the experiment process and discuss the results. Limitations and the future work for the project will be pointed out as well. Finally, the conclusion is given in Chapter 4.

# Methodology

This section will start by introducing the preparation work for the system, including constructing the legal concept tree and developing the sentence simplification module. Then, it will compare contextual phrases against legal terms to discuss if phrase generation algorithms are needed. At last, it will present a novel model for extracting legally important phrases from sentences.

## Preparation Work

We have created two standalone modules that support and improve the main algorithm. Both of the modules have been open-sourced on [GitHub](https://github.com/hkulyc/legalAI).

**Legal Concepts Tree** contains more than 13000 common legal terms used in Hong Kong Judiciary. It is adapted from the legal taxonomy developed by Sweet & Maxwell. As shown in Figure , it has a perfect tree structure where parent terms are more general than their children. The tree is optimized to store in a hashed map for faster key search. With this tree, we can use keyword matching to identify the legal terms in the sentence. If necessary, it is also useful for substituting the legal terms with more general ones.

**Sentence Simplification** is the task to split complex or compound sentences into simple sentences, each of which has only one pair of subject and verb. There are previous works on this task using constituency parsing, such as Garain et al. and Das et al. . We implement and improve the algorithm by Garain et al. to fit into the latest Stanford Constituency Parsing rule . This implementation is also intelligent to skip the proper names, such as "Fish and Chip" or "Johnson and Johnson". This module can help to simplify long sentences, which are common in court judgements, so that operations afterward can be more accurate.

## Tag Format Comparison

One critical question to ask is that since the legal concept tree can help to find legal terms, is it necessary to take one step further to construct contextual phrases based on the legal terms? For instance, given the sentence "She suffered a fracture of the superior and inferior public rami of the right pelvis.", is tag "fracture" enough, or is tag "fracture of rami of pelvis" better? While phrase tags can contain more local contexts and information than word tags, they may sometimes be broken during generation. It is impetuous to draw any conclusion at this point, so one experiment is set up to test this question. The result is shown in chapter 3. The remaining sections in chapter 2 will focus on generating phrase tags only.

## Main Algorithm

As stated in 1.4, Aspect-driven Topic Modelling will be used and the most relevant sentences to the aspects will be found first. The main algorithm solves the problem of extracting legally important phrases from each sentence.

In step A, the Stanford Enhanced Dependency should be used. It differs from the Universal Dependency in containing prepositions inside relations. It will benefit the reconstruction of extracted words in step C. It should be noted that the relations starting with "acl:" should be eliminated as they will cause dependency cycles. Step B finds the shortest subset of the sentence that contains the matching legal term. Based on Constituency Parsing, it is to find the shortest component with a tag that ends with "P", such as "NP", "VP". This will include the neighbor words of the legal term to make the result more specific. For example, given the constituency tree in Figure 2.3, assume that the word "condition" is found to be legal term.

Step B finds the smallest component "His mental condition". Without this step, the word "condition" itself might be too general and ambiguous.

Step C is to extract faraway related words with respect to the matching legal term using Dependency Parsing. Starting from the legal term w, the program will recursively find the parent of it in the dependency tree until the root is reached or a verb is met. Since the verb is connected to its subject using relation "obj", the condition therefore becomes meeting "obj". During the backtracking, the prepositions and conjunctions in the dependency relation should also be recorded. They are stored after ":" in the relation within the set "nmod:", "obl:", "conj:", "advcl:". Finally, the words in the reconstructed phrase should have the same order as the original sentence. For the example of Figure 2.3, starting from the word "condition", "unstable" is found to be the first parent and "bad" is the second. There is a conjunction word "and" between the "bad" and "unstable", according to the relation "conj:and", so "and" should be inserted in between. As a result, the tag "His mental condition bad and unstable" is generated.

Overall, this algorithm is a hybrid approach that utilizes both Constituency Parsing in extracting neighboring phrase and Dependency Parsing in finding faraway relational words. To make the tags more fluent and natural to people, we also added several heuristic operations to the algorithm.

1.  If the last word of the recursive step C is a link verb, such as "is", "are", we should truncate it.

2.  If the smallest component found in step B does not help specify the matching term, for example only adding "the" in front of the matching term, step B should be skipped. In the implementation, the skipping condition is that the length of the component is two and the POS tag for the first word belongs to {'DT', 'IN', 'PRP\$', 'CD'}.

3.  Sometimes, the preposition "to" before a verb cannot be successfully recognized by the Dependency Parsing. We should add "to" in front of the word with POS Tag "VB".

## Similar Documents Recommendation

Assume we have obtained the tags, ultimately the user will choose a few of them to navigate to other judgements which may mention similar legal concepts. According to the user survey, users often have a few interested legal concepts in mind during searching. Therefore, this aspect-specific similar document matching may be more useful than comparing two judgements by the full text. To solve the problem, the simplest way is to directly invoke the default search engine and use the selected tags as search input. This way, the selected tags are compared with the whole passage of other documents. Another way is to compare the selected tags with tags from other documents. The theoretical basis of this approach is that the tags have summarized most aspects of a court judgement (Requirement 4 in 1.4). Narrowing the searching scope to tags only can help to eliminate interference from other unimportant texts, making the recommendation more accurate. The quality of this recommendation will be tested in the following experiment. Algorithm shows the details of the recommendation algorithm by comparing selected tags with tags from other documents.

Basically, the similarity between each selected tag and another document is equal to the maximum normalized inner product of all possible tag combinations. The similarity score between all selected tag and another document is the sum of such similarities.

# Evaluation

This section detailly describes the experiment to evaluate the quality of the tags generated. Also, the experiment results will be revealed and the future work for the project will be pointed out.

## Experiment setup

As mentioned in section 2.3, the main algorithm for generating phrase tags is a hybrid approach of Dependency Parsing and Constituency Parsing. To test whether the use of both parsing techniques is necessary, we set up a comparison experiment. For each court judgement, four tag sets are generated using 1. only Dependency Parsing 2. only Constituency Parsing 3. the Hybrid Approach 4. Word tags only. Each tag set starts from three starting sentences derived from Aspect-driven Topic Modeling. Each document along with all the resultant tags will be distributed to at least two participants to minimize personal bias. Participants are asked to select the tags that fulfill all three requirements in 1.4. After getting the result from this experiment 1, we continue to test the tags format issue mentioned in section 2.2. The recommendation abilities of word tags and phrase tags are tested in this experiment 2. For each judgement, participants will be provided with the top 5 recommendations from word tags and phrase tags respectively in pairs. Participants are asked to find the better recommendation, which is more similar to the original judgement in each pair.

## Experiment Results

We invited 7 participants to evaluate on tags of 15 independent court judgements. Table shows the results of experiment 1. The average number of tags generated by the four approaches per judgement and the percentage of good tags are provided. In total, 6.8\*15 = 102 tags are generated by the hybrid method, and 1.76\*15 = 26.3 of them are good.

| Methods                   | \# of tags | \# of good tags | Good tags rate |
|:--------------------------|:----------:|:---------------:|:--------------:|
| Only Dependency Parsing   |     8      |        1        |       0        |
| Only Constituency Parsing |     5      |        1        |       0        |
| Hybrid approach           |     6      |        1        |       0        |
| Word tags                 |     10     |        1        |       0        |

Statistics of good tags per judgement

From the statistics, we can conclude that the hybrid approach generated the greatest number of good tags for each judgement (1.76). Also, although the constituency parsing approach generated the least good tags (1.56), the percentage of the good tags is the highest (0.268). Considering the purpose of the model, which is to generate tags for the users to choose, we should prefer the absolute number of good tags to the rate of it. Therefore, both Dependency Parsing and Constituency Parsing is necessary. The hybrid approach is the best phrase generation method. As for the word tags, the number of good tags is less than phrases, and the rate of good tags is the lowest. As a result, legal terms are not comparable to contextual legal phrases in terms of fulfilling the requirements of tags.

| Cases              | \# of pairs | Rate |
|:-------------------|:-----------:|:----:|
| Phrase tags better |      0      |  0   |
| Word tags better   |      1      |  0   |
| Both are good      |      0      |  0   |
| Neither is similar |      1      |  0   |
| Total              |      5      |  1   |

Statistics of better recommendations per document

Table shows the results of the recommendation ability of word tag and phrase tag in experiment 2. Since 5 pairs of external documents are recommended for each judgement, there are 15\*5=75 pairs of comparison tested. Here, we use the hybrid approach for the phrase tag generation.

From the table, we can find that the frequency of word tags being better is higher than the number of phrase tags being better. This indicates that there is still room for improvement for the recommendation algorithm based on phrase tags. The possible direction will be discussed in section 3.4.

## Limitations

The percentage of good tags from the hybrid approach is around 26%. Although the purpose of the task does not require tags to have an extremely high acceptance rate, higher rate can definitely resolve some misunderstandings and improve user experience. Based on the feedbacks from experiment participants, the reason of bad tags mostly owes to not fulfilling requirement 2 from section 1.4. They often present broken or confusing meanings. The root of this problem is obvious. Both Dependency Parsing and Constituency Parsing are syntactic parsing, indicating that they do not consider semantic information enough. For example, the correct punctuation of sentence "Drive the lorry for loading and unloading goods" should be "Drive the lorry for loading goods" and "… for unloading goods". However, the constituency parsing splits like "Drive the lorry for loading" and "… for unloading goods". The parsing simply finds the result with the highest probability in grammar regardless of the obvious parallelism of antonyms: "loading" and "unloading". Therefore, to improve the main algorithm, essentially a new parsing method that maximizes both grammatical and semantic probability is needed. Furthermore, since the main algorithm starts from most relevant sentences of each aspect, the result of AdTM will affect the final performance to large extent. Therefore, it is also necessary for the algorithm to find better document summarization methods alternative to AdTM that can elevate the upper limit.

## Future Work

One promising way to improve the recommendation based on phrase tags is giving weights to different tag combinations. As illustrated in 2.4, we are summing over the maximum of similarities of combinations by considering all tags equal. However, different tags come from different sentences that belongs to topics in four different aspects. If two tags come from the same aspect or even the same topic, their similarity score should be assigned a higher weight. For example, two tags both describing the losses of the personal injury cases should be naturally closer than two that describes different aspects. What's more, currently training sets and testing sets are all court judgements of Personal Injury cases. The effectiveness of this system on other case types is not tested. Future work should focus on applying to model to various types of cases. Finding more applications for the system is also an interesting topic. This model is designed for the HKLII website. However, after a simple modification of the Legal Concept Tree in section 2.1 to fit into local context, this model can be applied to other law systems. Furthermore, other professional areas that have similar concept tree structures can easily adapt the model to their advantage.

# Conclusion

This report introduced the model that labels court judgement. The system uses Aspect-driven Topic Modeling as the starting point. The main algorithm in the system that extracts phrase tags from a sentence is a hybrid method of Dependency Parsing and Constituency Parsing. Experiment results show that these tags outperform general legal terms. Although the aspect-specific recommendation is more useful than full passage comparison, it sometimes did not recommend the most similar documents compared with the benchmark algorithm. Future work should focus on the improvement of the recommendation algorithm for phrase tags, the seeking of alternative system components and the discovery of more use cases for the system.