author: Sebastian Eibl$^1$, Ulrich Rüde$^{1, 2}$
bibliography: literature.bib
title: A Modular and Extensible Software Architecture for Particle Dynamics

# Motivation

Modern simulation software frameworks must fulfill a variety of requirements. In a systematic design process, these requirements must be identified and formulated clearly in the preparatory stages of the development process. Only then a software design can be created that will fulfill as many requirements as possible. Here we will follow this approach in the design of a new simulation software for particle dynamics. Such a simulation software should be easy to use and it should be extensible by the end user. At the same time, high performance is desirable and a parallelization suitable for current supercomputers is necessary. Also, maintainability and portability are essential features since the software framework will often be used for a long time, exceeding the life span of computer systems. In this paper, we will discuss some of the requirements that we have identified for our software framework. We explain the implications and our design decisions to satisfy the requirements. In particular, we try to identify and modularize different components of particle dynamics simulations such that specialists can work on them independently. In the following, we present three technical requirements that guide the development of "Modular and Extensible Software Architecture for Particle Dynamics" (MESA-PD).

-   **flexible domain partitioning** Current state-of-the-art parallelization of particle dynamics software uses spatial domain partitioning. This domain partitioning is crucial for an efficient parallel execution. The way the domain is partitioned has impact on how well the workload can be balanced between the different processes and how much communication is needed during synchronization. Since different simulation scenarios will lead to different optimal domain partitionings, a flexible approach is needed to achieve good performance in all cases. A flexible domain partitioning is also important for coupled simulations which run in parallel. Ideally all coupled simulation components must share the same partitioning to avoid communication overheads. Therefore it is a great advantage when the domain partitioning can be easily adapted to that of other simulation modules.

-   **extensible particle data structure** Different interaction models between the particles require the particle to have certain properties. These can be material parameters, electric charge, temperature, and many more. Also, the framework user might want to store additional information when extending the functionality. However, all these possible properties will not be needed for every simulation. The general conventional implementation would have to provide all possible properties in the particle data structure. In most applications, however, only few features are used, therefore, many properties are unused. This leads to a potentially huge amount of memory that is wasted and that slows down the simulation if it has to be copied or sent over the network. Also, the maintainability of the particle data structure gets harder due to its size and dependencies. Here, a flexible approach is needed that allows to add and remove individual particle properties as needed for every specific simulation scenario.

-   **interaction model** MESA-PD is intended to be used as a scientific research tool. This makes it essential that particle interaction models can be adapted. It should be possible to implement new models easily within a short time to be able to test different approaches (*rapid prototyping*). In the best case, domain specialists who are not profoundly familiar with the framework should be able to modify existing or develop new interaction models. To achieve this, the interaction models must be decoupled from the rest of the framework as much as possible. Note, however, that this can easily lead to a design conflict regarding optimization. The possibility should still exist for a computer scientist to optimize the source code of the interaction model to achieve the best possible performance.

After identifying these requirements we will discuss our approach to fulfill them in Sec. –.

# Related Work

Different approaches were taken to create flexible, modular, performant and highly parallel software frameworks for particle simulations. The *LIGGGHTS*[^1] DEM software package uses input files in its own custom language to describe the simulation. It also offers the possibility to extend the software framework by writing own extensions. However, these extensions must be written in C++ and some understanding of the underlying structure of the software framework is needed. Similarly *GranOO*[^2] uses XML based input files to set up and run the simulation. The simulation is comprised of so called "PlugIns" which take care of different parts of the simulation. These PlugIns can be selected by the user via the XML file. It is also possible for the user to add custom PlugIns written in C++ or Python. These PlugIns access the simulation via a special API. *YADE*[^3] goes one step further and uses the scripting language Python for its input files. This allows the user the greatest flexibility in setting up the simulation as the full potential of Python and its packages can be used.

Another approach to let the user extend the software framework is shown by some molecular dynamics packages. They use high level languages like Python or their own embedded domain specific language (DSL) to describe the simulation. In order to achieve good performance on various hardware architectures they use just-in-time compilation to generate user specific executables for various architectures. However, to support MPI or CUDA additional wrapper libraries like pyMPI and PyCUDA are needed. Many packages using this technique claim that this can be done with almost no loss in performance compared to native C++ code. Packages that provide such capabilities with a varying degree of just-in-time compilation are for example *OpenMM* , *HOOMD-blue* , *MDL*  and *ppmd* .

# Our Contribution

The plugin model to extend an existing framework is limited by the API provided by the framework. This indirect access to the simulation data may be a limiting factor if data is needed that cannot be accessed via the API. Since the API cannot be extended by the plugin itself either workarounds must be implemented or the extension is not possible. Depending on the internal data structures of the simulation framework, the interaction may be slow if data has to be converted between the framework and the plugin.

In the MESA-PD design we therefore favor a single C++ codebase with no indirections. We use the waLBerla multi-physics framework  as a basis for the implementation of MESA-PD. waLBerla is an open source high-performance framework written in C++ that has shown excellent performance and scalability for Lattice Boltzmann  and particle dynamics simulations . Using this framework as a starting point, gives instant access to many utility functions like storing results into SQLite databases, writing vtk output files, etc. Additionally, advanced functionality like load balancing  can be used. We extend this framework with the new MESA-PD module which is the result of the requirements formulated in Sec. . Writing a native C++ module requires deep knowledge about the framework as well as profound programming skills. To achieve our requirements, however, we want the user to be able to make changes quickly. There are some proposals for the future C++ standard like reflections, metaclasses, and concepts which can make writing modules easier. Unfortunately they are not available in the current C++ standard. We, therefore, introduce an additional code generation step which generates parts of the module automatically. This additional step of compile-time transformations extends the possibilities the programmer has over bare C++ code. The code generation itself is performed via Jinja templates[^4]. Jinja templates allow us to introduce placeholders in the source code which will be filled later with the correct piece of code. But Jinja templates not only allow simple placeholder but also control structures like *if* and *for*. With these control structures one can disable replacements as well as generating a new line of source code for every item in a list. This alone, however, is not enough to make the coding much simpler. On top of the Jinja templates we introduce a Python library. The purpose of this Python library is to collect information about the simulation the user wants to conduct in a single place and then forward the information to all templates that work on that piece of information. In the end this allows the user of the framework to specify the simulation properties in a very high level representation. The distribution of the information is then handled by the Python library and the low level C++ source code is automatically generated by the Jinja template engine. This workflow is visualized in Fig. .

The code generation has to be run once by the user before the application gets compiled. With this approach we aim to leverage the full potential of a highly optimized framework and a unified code basis. Simultaneously, we can profit from the increased flexibility and straightforwardness gained from the code generation approach.

In the rest of this paper we will discuss the implementation satisfying the requirements presented. We will also point out where the new code generation approach greatly simplifies the programming task. A flexible domain partitioning approach using an interface is presented in Sec. . The handling of particle data and the extension with user supplied data is discussed in Sec. . Finally we present our kernel interface in Sec. . This allows to introduce new kernels without any deeper knowledge about the rest of the framework.

# Flexible Domain Partitioning

When running simulations in parallel with a spatial domain partitioning approach each process is responsible for a specific subpart of the simulation domain. All particle interactions within this region can be easily detected and resolved since the process has all necessary information. However, at subdomain interfaces, the available information is insufficient. Particles from neighboring subdomains might overlap with the local subdomain but information about the particle is not locally present. Typically this problem is solved by introducing *ghost particles*. Ghost particles are copies of particles located at other processes which do not belong to the subdomain of the process. In particle dynamics simulations where interactions occur as soon as particles collide, ghost particles are created when they overlap the subdomain. These ghost particles are created and updated by the synchronization algorithm. State-of-the-art synchronization algorithms, as described in , are independent of the actual subdomain geometry. The information they need is the ownership of a particle, i.e., which process stores the data, which is typically the process which handles the subdomain the geometrical center of the particle lies in. And additionally, if the particle overlaps adjacent subdomains which is commonly done by checking the bounding volume against the neighboring subdomains. So all tasks can be reduced to geometric functions that check points and bounding volumes against subdomains. As long as one can provide these functions the synchronization algorithm can be implemented without specific knowledge about the domain partitioning. For this purpose MESA-PD introduces an interface to the domain partitioning that covers all functionality that is needed for an efficient parallelization of the simulation. Thus the algorithms become independent of the domain partitioning. Different implementations of this interface can take care of various peculiarities of the simulation carried out. Three exemplary domain partitionings realized with this interface can be seen in Fig. . Within the waLBerla multi-physics framework an implementation for the distributed forest of octrees domain partitioning  is available. This allows an easy interaction with additional waLBerla modules like the Lattice Boltzmann Method implementation within a single application . The interface is also implemented by the HyTeG finite elements multigrid framework which uses unstructured tetrahedral meshes for the domain partitioning . An artificial domain partitioning into spherical shells is also shown in Fig. .

# Particle Data Structure

A flexible particle data structure that can be adapted according to the needs of the current simulation scenario is important to keep the memory footprint small and performance high. However, changing the properties of the particles also involves adapting some algorithms. For example, the algorithm which updates all ghost particles needs to know what particle properties it should synchronize. To simplify the process of adding and removing particle properties and to update all algorithms accordingly we use our additional code generation library. The library offers high level functions to add properties to the particle data structure. This greatly reduces the workload for the programmer and it is also less demanding on the programming skills. A new particle property can be added in the following form:

``` python
addProperty(name, datatype, defValue, syncMode)
```

`name` names the property and `datatype` specifies the data type of the property. The data type can be any valid C++ data type. The `defValue` is the value the property gets initialized with when a new particle is created. The most interesting parameter for parallel simulations is the `syncMode`. This parameter controls when and how this property gets synchronized between different processes. Different modes are available, namely:

**NEVER** This property gets never synchronized. This is useful if it is used only to store intermediate results.

**COPY** This property is copied exactly once when a new ghost particle is created. This is typically used when the property does not change but is different for every particle. Depending on the simulation this can be something like *mass*, *particle radius*, etc.

**MIGRATION** Properties annotated with this syncMode are synchronized when the ownership of a particle changes (i.e. the particle leaves the current subdomain and is now in the subdomain of a different process). This can be used for example to synchronize the old force in a velocity verlet integration scheme.

**ALWAYS** This property is synchronized in every iteration. This is used for properties which change frequently like *position*.

During the code generation, the library passes this information to the templates that need it. The templates are then translated into C++ source code. In the following we want to illustrate this process. First, the user defines that the particle data structure should contain three properties: position, radius and force. The position should be synchronized in every synchronization step whereas the radius only needs to be synchronized when a new (ghost) particle is created. The force is recalculated in every time step and is therefore never synchronized. All this information is specified with the following three lines:

``` python
addProperty("Position", "Vec3", "0,0,0", "ALWAYS")
addProperty("Radius", "double", "0", "COPY")
addProperty("Force", "Vec3", "0,0,0", "NEVER")
```

This information is then forwarded to all source code templates. An exemplary template that packs the particle `particle` into a buffer looks like this:

``` python
{%- for prop in properties %}
{%- if prop.syncMode in ["COPY", "ALWAYS"] %}
buf << particle.get{{prop.name}}();
{%- endif %}
{%- endfor %}
```

The for loop prints the enclosed source code for every property into the C++ file. Additionally, the if statement selects only specific properties that are needed in this context. With the information provided by the user the template gets expanded into:

``` c++
buf << particle.getPosition();
buf << particle.getRadius();
```

One of the major benefits of this approach is that the user only has to specify the particle properties once. According to what the user specified, multiple source files as well as many occurrences within one source file are adapted. This greatly reduces the burden of remembering all places in the source code which need to be adapted to work with this particular set of properties. It is also less error prone since pairs like packing and unpacking are both generated. This eliminates possible inconsistencies. But not only all algorithms are adapted automatically also the user gets exactly the particle data structure which perfectly suits the scenario. However, one additional step is needed. Before compiling the application code the user has to run the code generation once. After that the application can be compiled like usual.

# Particle Interaction Models

In this section, we discuss how particle interactions can be implemented in MESA-PD. Saunders et al.  used the fact that most operations in their molecular dynamics code are carried out either on every molecule (apply gravity, do time integration) or on every pair of molecules (interactions). They proposed to separate these operations into individual functions which they called *kernels*. With this approach one can isolate the interaction models from the rest of the code. The idea that all operations concerning molecules can be written as kernels can also be applied to general particle dynamics codes. Using this concept, a domain specialist can implement an interaction model by writing a function which takes two particles as input and calculates the interaction. In our approach we go even further and also decouple the kernel code from the actual data. We use a so called *accessor* interface that maps between the kernels and the actual data structure. When a kernel accesses particle properties it does so via the accessor. The accessor than locates the data and passes it to the kernel. The accessor is used for reading as well as writing particle data. This allows to switch the implementation of the accessor interface without touching the kernels. With this approach one can change the particle data structure independently of the kernels. Only the implementation of the accessor interface has to be adapted.

``` c++
template <typename Accessor>
inline void EulerIntegrator::operator()(const size_t idx,
                                        Accessor& ac) const
{
   ac.setPosition(idx, ac.getInvMass(idx) * ac.getForce(idx) * dt_ * dt_ +
                       ac.getLinearVelocity(idx) * dt_ +
                       ac.getPosition(idx));
   ac.setLinearVelocity(idx, ac.getInvMass(idx) * ac.getForce(idx) * dt_ +
                             ac.getLinearVelocity(idx));
   ac.setForce         (idx, Vec3(0, 0, 0));
}
```

This approach comes with many benefits. First of all, domain specialists who write the kernel code can do so without worrying about where data is stored and how to access it. All kernel accesses to the outside world are represented as *give me this information* and *store that information*. This way the kernel code is completely independent of the rest of the framework. It is also possible to use the kernels with a different framework as long as the data structures of the framework can be accessed via a particle accessor interface making the kernels widely usable. This greatly increases the flexibility of the kernels and also offers more possibilities in coupling different simulation frameworks.

# Conclusion

In this paper we presented a new approach to extend the high-performance framework waLBerla with a new particle dynamics module MESA-PD. MESA-PD employs a code generation step to simplify the task of writing modules for highly optimized frameworks. This additional step is realized by using a combination of Jinja templates and a newly designed Python library. With this approach, the user can give a high level description of the simulation using the Python library. In a second code generation step C++ source files are created by the Jinja template engine using the information the user has provided. This way the C++ source code files are tailored exactly to the description the user has given.

For the design of the module we have identified requirements that are essential for a modern particle dynamics framework. We then presented our resolution of these requirements within our newly developed module. The new module allows a more flexible domain partitioning in parallel simulations. This simplifies the task of coupling simulations as well as experimenting with more efficient domain partitionings tailored for a specific situation. We also introduced an advanced approach to create individual data structures for every simulation without manual code rewrites. Finally, we showed our approach to decouple the code for particle interactions not only from the rest of the framework but also make it independent of the data structures used to store the properties of the particles.

The new design has many benefits for the user of the framework. The code generation approach greatly reduces the lines of code the user has to write himself. If a single piece of information is needed at multiple places throughout the source code the code generation takes care of adapting all files accordingly. For example, after defining a particle property with just one line in Python the correct packing and unpacking functions to MPI buffers, debug output to the terminal, vtk output, output to databases, etc. are generated automatically. It not only saves time for the user it also ensures that there are no inconsistencies between the functions which could possibly lead to hard to track down errors.

The strict separation of data and kernels via the accessor interface allows to change parts of the code without interfering with other parts. With this approach specialists do not have to know the whole code base to introduce their knowledge. Additionally, due to the clear separation of all source code parts they might also be transferable to other frameworks.

[^1]: https://www.cfdem.com/

[^2]: https://www.yakuru.fr/granoo/index.html

[^3]: https://yade-dev.gitlab.io/trunk/

[^4]: http://jinja.pocoo.org/


abstract: Although machine learning has become a powerful tool to augment doctors in clinical analysis, the immense amount of labeled data that is necessary to train supervised learning approaches burdens each development task as time and resource intensive. The vast majority of dense clinical information is stored in written reports, detailing pertinent patient information. The challenge with utilizing natural language data for standard model development is due to the complex and unstructured nature of the modality. In this research, a model pipeline was developed to utilize an unsupervised approach to train an encoder-language model, a bidirectional recurrent neural network, to generate document encodings; which then can be used as features passed into a decoder-classifier model that requires magnitudes less labeled data than previous approaches to differentiate between fine-grained disease classes accurately. The language model was trained on unlabeled radiology reports from the Massachusetts General Hospital Radiology Department (n=218,159) and terminated with a loss of 1.62 and a word prediction accuracy of 62%. The classification models were trained on three labeled datasets of head CT studies of reported patients, presenting large vessel occlusion (n=1403), acute ischemic strokes (n=331), and intracranial hemorrhage (n=4350), to identify a variety of different findings directly from the radiology report data; resulting in AUCs of 0.98, 0.95, and 0.99, respectively, for the large vessel occlusion, acute ischemic stroke, and intracranial hemorrhage datasets. The output encodings are able to be used in conjunction with imaging data, to create models that can process a multitude of different modalities. The ability to automatically extract relevant features from textual data allows for faster model development and integration of textual modality, overall, allowing clinical reports to become a more viable input for more encompassing and accurate deep learning models.
author: Neil Deshmukh
Moravian Academy \| Lehigh University; Bethlehem, PA 15018
`firstname.lastname@example.com`

; ; Selin Gumustop^1^
`firstname.lastname@example.com`

; Romane Gauriau^1^
`firstname.lastname@example.com`
; Varun Buch^1^
`email@example.com`

; Bradley Wright^1^
`firstname.lastname@example.com`
; Christopher Bridge^1^
`email@example.com`
; Ram Naidu^1^
`firstname.lastname@example.com`

; Katherine Andriole^1^
`email@example.com`; Bernardo Bizzo^1^
`email@example.com`

^1^MGH & BWH Center for Clinical Data Science; Boston, MA 02114
bibliography: bibliography.bib
title: Semi-Supervised Natural Language Approach for Fine-Grained Classification of Medical Reports

# Introduction

Electronic Medical Record (EMR) data is the current standard system for the collection of medical information about a patient, and contains the vast majority of dense clinical information. The number of practices adopting an EMR system has been steadily increasing over the last few years (see Fig. 1), and has become a staple within healthcare facilities . There is an incredibly large amount of data within these systems, allowing for incredible insight with machine learning techniques, especially through textual data. Clinical reports from various sources, such as radiology exams, are commonly stored in the EMR systems, encompassing not only a comprehensive representation of the patients' status, but an accompanying analysis by the medical practitioner themselves, embedding an untapped clinical inference within these documents .

The cohort generation process for the development of datasets for machine learning applications requires the selection of appropriate data, based on specific criteria. Among several methods of such cohort selection for medical imaging data, existing radiology reports generated for clinical interpretation can be used to identify exams containing findings of interest. The radiology reports are commonly manually reviewed and annotated by experts to generate cohorts for medical imaging machine learning models, which is a time-consuming process that can be a bottleneck for development.

Extracting this clinical information from within the vast amount of medical documents has been an immense challenge, due to the significant variations in medical terminology, and the complex, unstructured nature of textual data. Practitioners display a wide array of variability, which makes it difficult to utilize keyword searches or any form of supervised training, as the labeling process of such a large amount of data would be a monumental effort. Standard text encoding processes are unable to fully understand the contextual relationship found in medical reports .

In terms of the analysis for this initial research, acute strokes were the focus, due to their impact on the general populace. There are 795 thousand acute strokes in the US annually, and, taking more than 140 thousand lives per year, they are the third leading cause of death . Out of this immense number of strokes, 87% are ischemic, in which blood flow to the brain is blocked . As such, evidence of hemorrhage contraindicates IA/IV treatment , so it is relevant to future treatment options; 38.7% of the subset of acute ischemic strokes presents large-vessel occlusion . Overall, acute ischemic stroke is a leading cause of long-term and preventable disability, all the while costing the US an estimated \$34 billion each year.

The increasing prevalence of EMR data through healthcare systems made it imperative to develop systems to automatically learn encodings of complex textual data. The purpose of this research was to develop an algorithm to automatically classify head CT radiology reports for specific stroke findings.

# Methods

## Database Preprocessing

The entire text corpus utilized in the development of the language model was aggregated from the Massachusetts General Hospital (MGH) Department of Radiology archive. The dataset consisted of 218,159 radiology reports detailed by medical practitioners. The dataset was split into training, validation, and testing sets in a ratio of 85/15/5.

Words that showed up less than five times in the dataset were excluded from the vocabulary, and were usually typos in the reports or specific proper nouns. The dataset was also tokenized through the spaCy format to allow for better feature extraction by the model, utilizing tokens for uppercase characters and repetition. Each word input is set as a one-hot encoded vector as an input to the language model, corresponding to a vocabulary index and word embedding in the initial layer.

It should be noted that there is a clear distinction between the general unlabeled dataset, which is utilized for the unsupervised encoder training, and the classification dataset, explained below, which contains labels and is utilized for the supervised decoder training.

The labeled datasets were collected through hand-labeled reports by expert clinicians, who parsed through each report individually, and identified the specific labels necessary for classification.

The development set ratios were determined by the number of data points in the set. The labeled datasets for large vessel occlusion (n=1403), acute ischemic stroke (n=331), and intracranial hemorrhage (n=4350), had development set splits of 30/20/50, 30/10/60, and 10/20/70, respectively. The same text preprocessing for the language model occurs for the inputs in the classification set. The output for the acute ischemic stroke model is binary (presence or absence of acute ischemic stroke); meanwhile, the class outputs for the multilabel large vessel occlusion model are: M1 (first segment of the middle cerebral artery), M2 (second segment of the middle cerebral artery), distal ICA (internal carotid artery), and Basilar; and the class outputs of the multilabel intracranial hemorrhage model are: SDH (subdural hemorrhage), SAH (subarachnoid hemorrhage), IVH (intraventricular hemorrhage), IPH (intraparenchymal hemorrhage), and EDH (epidural hematoma).

## Data Exploration

Even from preliminary analysis, the dataset of patient reports contained great variability in the expression of conditions. The amount of dense information within each report was difficult to parse, as doctors embedded many diagnoses and negatives within a single paragraph.

Analyzing the most frequent n-grams within differing classes also did not identify a clear pattern, as a simple keyword search would be unable to contextualize the report as a whole, including the negation of particular conditions. The classes, even if simplified to the base positive and negative cases for a medical anomaly, contained few defining characteristics indicative of a classification.

Shorter results tended to be a negative label, but, in some instances, the impression had descriptions of many different conditions, and with negation challenging to detect, it would be extremely difficult for the model to interpret without understanding context.

As part of the data exploration process, a scatter plot utilizing Term Frequency Inverse Document Frequency (TFIDF) was generated. The TFIDF value of a word is generally defined as:

$$tfidf( t, d, D ) = tf( t, d ) \times idf( t, D )$$

The term-frequency term refers to the number of instances a term is found in a document, while the $idf$ term, demonstrating its inverse proportionality to the number of documents it is found within, is more specifically defined as:

$$idf( t, D ) = log \frac{ |  D  | }{ 1 + | \{ d \in D : t \in d \} | }$$

As TFIDF vectorizers generate sparse vectors, Singular Value Decomposition (SVD) was used to reduce the dimensionality of the data. Unfortunately, as was identifiable by viewing the plot, the TFIDF clusters did not seem to be separable. The TFIDF encodings were explored further, utilizing the cosine similarity function to view similarity between documents, defined as:

$$cos( x,  y) = \frac { x \cdot y}{|| x|| \cdot || y||}$$

After many iterations of report exploration, it was determined that the TFIDF values of documents were not a helpful metric in the clustering of similar reports, as there seemed to be no visible pattern in the reports embedded near one another in the high dimensional space of the TFIDF encodings.

Through this preliminary data analysis, it was identified that it would be difficult for a model to determine relevant encodings and generate classifications without the ability not only to understand the general context, but also the specialized medical vocabulary found in the dataset. More specifically, due to the magnitudes greater unlabeled data available, a model that would be able to generate relevant encodings utilizing an unsupervised approach would allow for scalability and ease-of-use in downstream applications; this was the goal of the project.

## Machine Learning Approaches

While the classification of the medical reports found in the dataset seemed too fine-grained a task to classify for standard ML approaches, they were still attempted with the binary occlusion dataset to obtain a preliminary measure of performance. TFIDF vectorization was utilized as the preliminary input into statistical ML models: focusing on Naive Bayes, logistic regression, and Support Vector Machine (SVM). While a variety of other ML approaches were utilized, including random forest and k-nearest-neighbors, they did not achieve any significant results compared to the three analyzed further. The TFIDF models all performed worse than random classification (0.5).

A purely statistical model seemed unable to understand context in the impressions, specifically with variable descriptions of similar medical phenomena, as it is difficult to extract fine-grained context from frequency scores. To provide the models with embedding inputs that contained high dimensional structure, and some contextual embeddings, a Bi-directional Encoder Representations from Transformer (BERT) model was transfer trained on an unlabeled set of reports (n=218,159). The large, cased BERT model subtype with embeddings generated from the concatenation of the final four layers performed optimally in testing. The encodings generated by the custom BERT model were used to vectorize the text as an input for the standard machine learning models described earlier.

While the models did marginally better with BERT-embeddings, both the inability to pre-train the BERT model due to computational limitations, and the impairment to create fully context-based embeddings, due to the mask-based training structure, makes it difficult for fine-grained classification tasks.

In viewing that there seemed to be some context and language understanding within the BERT embeddings, a more complex fully connected, neural network (NN) was tested as the classifier, compared to the previous statistical approaches. The hope was that the NN would be able to identify more fine-grained features than a purely unsupervised network. The best architecture was one that was four layers of 512, 256, 128, to output, while L2 regularization and dropout were both used. The NN classifiers' performance was significantly higher compared to other previous approaches.

|              Model              | F1-Score | AUC  |
|:-------------------------------:|:--------:|:----:|
|       Naive Bayes (TFIDF)       |   0.28   | 0.37 |
|        Logistic (TFIDF)         |   0.35   | 0.43 |
|           SVM (TFIDF)           |   0.33   | 0.42 |
|     Neural Network (TFIDF)      |   0.53   | 0.46 |
|  Naive Bayes (BERT-Embedding)   |   0.39   | 0.46 |
|    Logistic (BERT-Embedding)    |   0.41   | 0.68 |
|      SVM (BERT-Embedding)       |   0.44   | 0.39 |
| Neural Network (BERT-Embedding) |   0.61   | 0.72 |

ML Model Results

## Development of Unsupervised Language Model

While the NN improved the metrics compared to previous approaches, they were still not optimal for cohort generation or report classification.

As mentioned, the main challenge in processing medical reports is the variability in the description of diseases, making it difficult to classify with a small sample set. Techniques that are currently widely used, identify specific keywords through an iterative process, which is time-consuming and is not capable of accounting for the variability and complexity found in medical reports. Approaches that utilize models that do not understand contextual relations in the text are not capable of easily generalizing to different datasets. There was a need to optimize the cohort generation process to reduce the resources and time necessary for development .

For these reasons, an unsupervised approach was utilized, simplifying the learning process by allowing the model to identify encodings, medical terminology, and context automatically, without any external work for a wide array of records. This allows for great scalability, simply by retraining the model on a larger corpus, were one to become available. An encoder model with the ability to generate single-size encodings is a powerful tool to create a high-dimensional representation of reports that are useful in a variety of downstream tasks.

The model architecture is a Recurrent Neural Network (RNN), which is specifically well suited to process textual data, due to their self-inputting nature, allowing for the detection of specific patterns in varying intervals of data. A many-to-one approach was utilized in this model, as a sequence was the input, and a word prediction classification was outputted. Both Gated Recurrent Units (GRUs) and Long Short Term Memory (LSTM) units were tested, and the LSTM networks resulted in a higher accuracy (see Fig. 3). The model is bidirectional, allowing for better context analysis from intervals of data.

The Pytorch Machine Learning Library , was used to develop, train, test, and optimize the neural network (NN). The RNN has a structure containing five layers and 31,683,132 trainable parameters. The model's preliminary embedding layer contains 28,362 parameters.

This model was determined after a period of research, as it allows for entirely unsupervised natural language understanding, specifically for medical reports. Processing each next word as the 'labeled' dataset transforms the unsupervised approach of language modeling into one that has corresponding word labels that allow the model to learn encodings for the report document into a lower-dimensional representation that simplifies the classification for the densely connected network. Through the process of self-identifying features, the model is capable of learning the context of medical terminology and developing encodings of the model that are relevant for downstream machine learning approaches.

## Development of Classifier Model

In order to process the encodings from the language model, the classifier inputs are the activations of the model from the layer before the final word classification layer, with an encoding size of 400.

Due to the nature of the classifier head of the model, the RNN encoder can remain frozen while the classifier can be modified to accommodate any variety of inputs. The occlusion and hemorrhage classification models required a multilabel approach, with sigmoid activations on each of the corresponding output nodes to classes, as a patient could express multiple types of the conditions, while the stroke model was a binary classifier. Particularly, the models were trained with cross-entropy loss.

Both demonstrate the versatility of the language model, as if there are any clinical text machine learning analyses necessary, this model is capable of generating single-length representations of report documents that are high dimensional representations that allow for more efficient training pipelines.

## Training of Neural Networks

Once the NNs had been developed, they were trained solely on raw inputs, taking in the tokenized text without any handcrafted features. Utilizing a deep learning approach increases the usability of this model by allowing for this unsupervised approach. The RNN was trained to optimize for smaller matrix operations, to allow them to run inference on large datasets in a reasonable amount of time on a standard desktop system.

To emphasize, the RNN encoder model was trained on the unlabeled dataset (n=218,159) of radiology reports, and frozen when the optimal loss was achieved. The classifier models were then trained on top of the encoder, on the much smaller labeled datasets, for 500 epochs, after which the encoder was unfrozen to update parameters incrementally. Overall, it was trained over the course of 1500 epochs with cross-entropy loss and learning rate of 0.0003.

All of the previously mentioned training was conducted on a research computer, utilizing 32 GBs of GPUs, 16 vCPUs and, 20 GBs of RAM.

# Results

The language model RNN was evaluated with the testing loss, as accuracy is not an intuitive metric for optimization. The testing set (n=10,908) comprised 5% of the entire unlabeled report dataset. The final result was a loss of 1.62, with a vocabulary and embedding size of 18,362.

|  Dataset   | F1-Score | AUC  |
|:----------:|:--------:|:----:|
| Occlusion  |   0.96   | 0.98 |
|   Stroke   |   0.92   | 0.95 |
| Hemorrhage |   0.98   | 0.99 |

Semi-Supervised Model Results

The encodings from the second-to-last layer of the model were transformed to 3D space through Singular Vector Decomposition (SVD), an algorithm that identifies orthogonal values to transform high-dimensional representations by simplifying the data. (see Fig. 4)

Area Under the Receiver Operator Characteristic (AUROC) curve, which plots the true positive rate against the false positive rate, was the metric that was optimized to allow for thorough analysis, while F1-Score was also calculated in order to analyze the performance of the model further. In terms of both, a micro-averaging strategy was utilized to determine a singular metric across all classes. With a large number of classes, F1-Score allows a metric that optimizes to make sure that the algorithm identifies the classes correctly without many false predictions, specifically when the dataset is imbalanced. The models achieved AUCs of 0.98, 0.95, and 0.99 on the datasets on occlusions, stroke, and hemorrhages, respectively.

# Conclusion

The semi-supervised models and process outlined in this research achieved all the goals outlined and far exceed the current standard for medical record classification. By conducting completely automatic encoding of medical documents, across a wide array of subfields, this language model allows for the development of extremely well-performing models, simply utilizing magnitudes less labeled data. This scalable method results in a more efficient development pipeline for data curation, requiring a few samples to identify patients in a particular subset, classifying medical reports even if similar, through fine-grained classification. The language model, once trained, has a numerous amount of downstream uses, from text clustering, to cohort generation, to the generation of encodings as another modality input into standard ML algorithms. Overall, this model allows for the automated generation of encodings that contain relevant clinical information from medical report data, which has the potential to leverage an enormous set of patient data in the medical system, and lead to the development of more comprehensive models with a multitude of modalities.


abstract: Mathematics and its relation to the physical universe have been the topic of speculation since the days of Pythagoras. Several different views of the nature of mathematics have been considered: *Realism* - mathematics exists and is discovered; *Logicism* - all mathematics may be deduced through pure logic; *Formalism*: mathematics is just the manipulation of formulas and rules invented for the purpose; *Intuitionism*: mathematics comprises mental constructs governed by self evident rules. The debate among the several schools has major importance in understanding what Eugene Wigner called, ''The Unreasonable Effectiveness of Mathematics in the Natural Sciences.''$\,$ In return, this 'Unreasonable Effectiveness' suggests a possible resolution of the debate in favor of *Realism*. The crucial element is the extraordinary predictive capacity of mathematical structures descriptive of physical theories.[^1]

**The Reasonable Effectiveness of Mathematics in the Natural Sciences**
Alex Harvey${}^{a)}$
Visiting Scholar
New York University
New York, NY 10003

# Introduction

In an essay which appeared in his collection, *Symmetries and Reflections*, Wigner explored the connection between mathematics and science. He wondered that,

> ''$\dots$the enormous usefulness of mathematics in the natural sciences is something bordering on the mysterious and that there is no rational explanation for this.''

And, later,

> ''$\ldots$ the mathematical formulation of the physicist's often crude experience leads in an uncanny number of cases to an amazingly accurate description of a large class of phenomena.''

A similar view was expressed by Bochner[^2]

> ''What makes mathematics so effective when it enters science is a mystery of mysteries and the present book wants to achieve no more than explicate how deep this mystery is.''

And again, in Chapter 22 of the *London Philosophical Study Guide* .

> '' Mathematical reality is in itself mysterious: how can it be highly abstract and yet applicable to the physical world? How can mathematical theorems be necessary truths about an unchanging realm of abstract entities and at the same time so useful in dealing with the contingent, variable and inexact happenings evident to the senses?''

None mention the central mystery, which is the ability of an *appropriate* formulation to predict phenomena yet undiscovered. Indeed, as often noted, the principal problem in progress in physics is to find this formulation. It is our purpose to explore this ''mysterious''$\;$ relation between physical theories and their mathematical formulation and suggest an answer. Firstly an understanding of mathematics is essential.

# Mathematics

Mathematics arose from that most fundamental of mathematical activities - counting objects. In the most ancient of times, a person might *enumerate* objects of the same kind - say goats or sheaves of wheat and if he wanted to record the results, would have to invent a system of symbols to identify and differentiate two quantites. We can be sure that he would use the same symbol system to record different quantities of apples. He would quickly learn to add two collections of objects. This could well have happened anywhere groups of people progressed beyond being hunter-gatherers.

This simple skill evolved to the sophisticated arithmetic of the Babylonians . They exhibited great ingenuity in their ability to use tables of squares to solve various arithmetic problems and could solve quadratic and cubic equations. Similarly, a direct descent for spatial relations can be discerned. The Pharaonic Egyptians devised an elementary geometry for purposes of resurveying fields after the recession of the annual Nile floodwaters. This skill was elaborated to the extent that construction of temples and pyramids became possible. To accomplish all this, a unit of length was defined. In this way geometry was conflated with arithmetic. Despite their achievements, neither the Babylonians nor the Egyptians considered their mathematics to be more than a computational scheme. Their mathematics was devoted primarily to mensuration, computing taxes, keeping accounts and making astronomical calculations. They had no concepts of theorems or proofs. This was left to the Greeks.

The genius of the Greeks, beginning with the Pythagorean school was to discern an abstract coherent structure beneath the arithmetic of the Babylonians and Egyptians. They introduced the concept of *theorem and proof*. In this way, from the simple act of counting and the drawing of geometric figures, an elaborate system of mathematics was constructed in Classical Greece.

The Pythagoreans left no written record of their accomplishments. These are known largely through the writings of later Greek philosophers . They were besotted with number *per se* effectively founding number theory. But, it was their belief in the esoteric character of mathematics and its relation to the physical universe which is of interest here. They believed that all that could be known of the physical universe would be known through number. Of the Pythagoreans, Aristotle commented ,

> ''Contemporaneously with these philosophers and before them, the so-called Pythagoreans, who were the first to take up mathematics, not only advanced this study, but also having been brought up in it they thought its principles were the principles of all things. Since of these principles numbers are by nature the first, and in numbers they seemed to see many resemblances to the things that be modeled on numbers, and numbers seemed to be the first things in the whole of nature, they supposed the elements of numbers to be the elements of all things, and they exist and come into being - more than in fire and earth and water $\ldots$ all other things seemed in their whole nature to hold heaven to be a musical scale and a number.''

Many centuries later, Galileo put it more precisely (and famously).

> ''Philosophy is written in this grand book–I mean the universe–which stands continually open to our gaze, but it cannot be understood unless one first learns to comprehend the language and interpret the characters in which it is written. It is written in the language of mathematics, and its characters are triangles, circles, and other geometrical figures, without which it is humanly impossible to understand a single word of it; without these, one is wandering around in a dark labyrinth.''

Whether his view is Pythagorean, that is, does he imply an identity between the mode of description and the object described or that the mode and object are distinct, is not clear.

A similar sentiment, with clearer distinction, was expressed some three centuries later by Weyl in connection with the ''language''$\;$appropriate for application to gravitation theory.

> ''As everyone has to work hard learning language and writing before he can use them freely for expressing his thoughts, so here too the only way to shift the load of formulas from one's own shoulders is to be well acquainted with tensor analysis so that one can turn unimpeded by formalities to the true problems: Insight into the nature of space, time and matter insofar as they contribute to the construction of the objective reality.''

# Nature of Mathematics

For the scientist, mathematics is a tool; for the mathematician, it is an end in itself; for the philosopher, the philosophy of mathematics is, as Körner puts it,

> ''... not mathematics. It is a reflection on mathematics.''

The *nature* of mathematics has been debated by philosophers[^3] beginning with the Pythagoreans. They considered numbers to be inextricably connected with the physical world. Plato refined Pythagorean ideas to a scheme of *ideal* abstract structures mirrored in the immediate world as *discovered* through human activity. Subsequent modifications led to *Realism* in which the severe concept of ideals is modified. Simply put, mathematical structures have an independent existence and are uncovered by observation and ratiocination. This view has been held by many mathematicians such as Paul Erdös who often referred to an imaginary ''Book''$\,$ in which God had written down all the ''beautiful''$\,$ mathematical proofs. This was expressed more picturesquely by Everett

> ''In the pure mathematics we contemplate absolute truths which existed in the divine mind before the morning stars sang together, and which will continue to exist there when the last of their host shall have fallen from heaven.''

A more temperate exposition of the same view was expressed by the distinguished British mathematician, Hardy ,

> ''I have myself always thought of a mathematician in the first instance as an observer, who gazes at a distant range of mountains and writes down his observations.''

In the late 19th Century, a radically new approach by Frege , subsequently termed *Logicism* claimed that all mathematics was just a logical structure.

> ''Arithmetic thus becomes simply a development of logic, and every proposition of arithmetic a law of logic albeit a derivative one. To apply arithmetic in the physical sciences is to bring logic to bear on observed facts; calculation becomes deduction. The laws of number will not, $\ldots$, need to stand up to practical tests if they are to be applicable to the external world; for in the external world, in the whole of space and all that therein is, there are no concepts, no properties of concepts, no numbers. The laws of number, therefore, are not really applicable to external things; they are not laws of nature.''

A modified view was earlier given by Kronecker .

> ''God created the numbers, all the rest is the work of man.''

This avoids the extreme difficulty of establishing the system of numbers by purely logical arguments encountered later by Whitehead and Russell in their *Principia Mathematica* . They do not establish that $1+1=2$ until page 362 of Volume 1.

The concept of *Formalism* is due to Hilbert who described it simply as[^4] .

> ''Mathematics is a game played according to certain simple rules with meaningless marks on paper.''

Wigner, early in his essay, in the section titled *What is Mathematics?*, provides the same view of mathematics.

> ''$\ldots$ mathematics is the science of skillful operations with concepts and rules invented for just this purpose.''

*Intuitionism*, created by the Dutch mathematician Brouwer , proposes that mathematical objects are mental constructions governed by self-evident laws.

In a class by itself is the school of thought that considers the brain to have been hard-wired with the number concept at birth[^5]

The various schools of thought have enabled intense debates among philosophers of science and mathematics. However, a conclusive argument strongly in favor of Realism can be constructed out the interaction of mathematics and physics, This will be described below.

# The Practice of Physics

The physicist attempts to understand physical universe. The endeavor is enormously facilitated through employment of Galileo's ''language of mathematics''. The use of mathematics entails representing each physical entity of interest with a mathematical object. This is done by *mapping* these entities *uniquely* onto a *mathematical* concept. A velocity is a polar vector; a moment is an axial vector; time is a one-dimensional manifold; flat space-time is a four-dimensional manifold with a Minkowski metric. The process is a species of conceptual *isomorphism*. It can't be a homeomorphism if ambiguity is to be avoided. Once this mapping has been completed, the mathematics takes over completely; it has a life of its own. The rules of the particular mathematical structure govern. There is no longer any physics *per se*. At the end of the calculation, the results are mapped back onto the physical universe.

At intermediate steps it should be possible to map the mathematics back onto the physics. That is, it should be possible to interpret every mathematical symbol with a physical construct. Dirac expressed it this way,

> ''The new scheme \[the representation of quantum states and variables\] becomes a precise physical theory when all the axioms and rules of manipulation governing the mathematical quantities are specified and when in addition certain laws are laid down connecting physical facts with the mathematical formalism, so that from any given physical conditions equations between the mathematical quantities may be inferred and vice versa. In an application of the theory one would be given certain physical information, which one would proceed to express by equations between the mathematical quantities. One would then deduce new equations with the help of the axioms and rules of manipulation and would conclude by interpreting these new equations as physical conditions.''

The ''Unreasonable Effectiveness''$\,$ is two-fold. Firstly, there is the incredible richness of phenomena which can be accurately described by a mathematical structure. Maxwell's equations furnish the means to accurately describe electromagnetic phenomena from power transformers to radio transmission. Einstein's equations of general relativity provide the basis for calculating gravitational phenomena from remarkably accurate solar system ephemerides to the gravitational radiation from pulsar PSR1913+16. The wealth of accurate description of atomic phenomena by the Schrödinger equation needs no elaboration. This brief list could be extended dramatically.

Secondly, and of decisive importance, the mathematics may describe phenomena completely unforeseen when the problem was originally formulated. think of the Dirac equation for the electron. What Dirac said about it later is precisely to the point.

> ''It was found that this equation gave the particle a spin of half a quantum. And also gave it a magnetic moment. It gave just the properties that one needed for an electron. That was really an unexpected bonus for me, *completely unexpected*.''

\[Emphasis added\] What he did not know at the time he published the equation \[1927\] was that, quite remarkably, it also described the positron which had yet to be discovered \[1932\].

The Schwarzschild singularity in the solution of the Einstein equations for an isolated point mass is another example. So was the yet to be observed radio waves in Maxwell's equations. Physics is full of these surprises. A less well known far more simple example is that of the exact solution of the simple pendulum. This is an elliptic integral , the real value of which corresponds to the given initial conditions. The imaginary period corresponds to a reversed gravitational field and an initial displacement of the bob which is supplementary to that in the original problem. The latter is an example of the principle that every prediction of a mathematical formulation of a physical has a ''real''$\;$significance.

# Reflections

The practice of physics is based on the concept that the physical universe, locally and globally, evolves in accordance with fixed rules. This is attested to by the consistent ability of independent observers to replicate each others results. It is this regularity which admits the use of mathematics to describe these results.

All of mathematics starts from a set of integers used for counting. The historical record shows that the ability to count evolved in different areas independently. The Babylonian and Mayan civilizations, among others, developed this skill. Contrarily, some cultures developed counting no further than "one, two, many." This suggests an objective independence of the integers and that they were *discovered* rather than created or, as suggested, that the number concept is hard-wired in the brain . Further, evidence provided by astrophysical observations implies that the laws of physics are invariant with respect to location and independent of era. This implies that the mathematical models created by extraterrestrials (if such exist) are identical, *modulo* symbology, to those here on earth. In turn, these imply the universal common existence of mathematics.

It was noted earlier that the relation between mathematics and physical phenomena is a species of isomorphism. The phenomena is mapped uniquely onto a mathematical structure and then mapped in the reverse direction. The extent to which the initial choice of mathematical structure is an accurate mapping is determined by how well the evolving physical phenomena remains isomorphic to the mathematics The relationship between mathematics and phenomena is reciprocal; *each is a representation* of the other. It is in this sense that mathematics has an objective reality. The problem of the physicist is to find that representation.

The identification of mathematics as objectively existent entails an interesting problem. Hawking has noted that mathematical structures are subject to the incompleteness theorems of Gödel:

> '' *... we and our models are both part of the universe we are describing.* Thus a physical theory is self referencing, ... . One might therefore expect it to be either inconsistent or incomplete. ... The theories we have so far are both inconsistent and incomplete.''

This is certainly correct but only of theoretical interest. There has yet to constructed mathematical models which are precisely descriptive of the phenomena we do observe. Despite an incredible accuracy of ten parts in a billion, the present formulation of quantum electrodynamics is notoriously lacking in rigor ; the ''standard model'', however accurate its predictions, is substantially more a protocol than a theory; a quantized version of the theory of general relativity has yet to be found. In brief, there is nothing yet in the mathematical models of observed physical phenomena to which Gödel's theorems might be applied. Moreover, our theories are certainly incomplete and will always remain so. With respect to the cosmos this is obvious. The strictures imposed by the principle of special relativity limits what information we can collect to the interior of the past light-cone. That is finite in extent, some 13+ billion years at the present time. The best that a physicist may hope for is a model which is both a) accurate in description b) possessed of predictive capability and c) mathematically rigorous. While the success of (a) and (b) have been substantial, this is not the case for (c). The approach to a final, mathematically rigorous ''theory of everything''$\;$is at best, asymptotic. Thus, for the finite future, Gödel's theorem is not a problem.

Physicists do not (with rare exception such as Newton's fluxions, the aforementioned Dirac delta function, or Heaviside's operational calculus) create any of the mathematics they utilize. The elaboration of existing and discovery of new mathematics is mostly the province of mathematicians. We thus have a very large apparatus of which only part is presently used by physicists. This suggested the question , ''Does any piece of mathematics exist for which there is *no application whatsoever in physics*?''$\;$The question has been addressed several times, most recently in 2001 . The answers were uniformly no. This, though suggestive, is not conclusive.

If either Formalism or Constructivism were accurate characterizations of mathematics then every mathematician could construct his own universe. There could be as many different universes as there were mathematicians engaged in the enterprise. The consequences of this are not easily described. The decisive evidence that Realism is the proper characterization is the *predictive* capability of the mathematical structures in which physical theories are couched. This strongly infers that mathematics and the reality it describes are both part of the objective universe and await discovery.

# Acknowledgments

Thanks: to (the late) Professor Arthur Komar for stimulating conversations; to Professor Hilail Gildin for useful information on the philosophy of mathematics and comments on the manuscript; and especially to Professor Engelbert Schucking who not only provided the translation of the quotation of Weyl, page 4, above but also read the manuscript critically. ${}^{a)}$Professor Emeritus, Queens College, City University of New York, firstname.lastname@example.com

[^1]: It is a pleasure to dedicate this paper to Josh Goldberg, long a valued member of the community of Generl Relativists and a good friend.

[^2]: It is noteworthy that Wigner was a Nobel Laureate in physics and Bochner, a distinguished mathematician. The talent for the two skills does not seem to be the same. Only a few gifted individuals such as Archimedes, Newton, or Gauss had supreme talents in both. This difference in talents can be observed in college upperclassmen. Expert knowledge and superior skill in the *use* of mathematics is the hallmark of the gifted physicist.

[^3]: For a concise introduction and useful bibliography, see the Wikipedia article, ''The Philosophy of Mathematics''. For a detailed introduction and discussion, consult Körner's ''*The Philosophy of Mathematics*''.

[^4]: Though widely quoted, the attribution of this comment to Hilbert is apocryphal.

[^5]: For an example of this approach see Dahaene .