abstract: Smart grid is a modified form of electrical grid where generation, transmission, distribution and customers are not only connected electrically but also through strong communication network with each other as well as with market, operation and service provider. For achieving good communication link among them, it is very necessary to find suitable protocol. In this paper, we discuss different hardware techniques for power monitoring, power management and remote power controlling at home and transmission side and also discuss the suitability of Zigbee for required communication link. Zigbee has major role in monitoring and direct load controlling for efficient power utilization. It covers enough area needed for communication and it works on low data rate of $20 Kbps$ to $250 Kbps$ with minimum power consumption. This paper describes the user friendly control home appliances, power on/off through the internet, PDA using Graphical User Interface (GUI) and through GSM cellular mobile phone.
author: N. Javaid, A. Sharif, A. Mahmood, S. Ahmed$^\sharp$, U. Qasim$^{\ddag}$, Z. A. Khan$^{\S}$
COMSATS Institute of IT, Islamabad, Pakistan.
$^\sharp$Mirpur University of Science and Technology, AJK, Pakistan.
$^{\ddag}$University of Alberta, Alberta, Canada
$^{\S}$Faculty of Engineering, Dalhousie University, Halifax, Canada.
title: Monitoring and Controlling Power using
  Zigbee Communications

# Introduction

n many countries, communication based controlling and monitoring architecture is used in smart grid to save power. Communication network may be wired or wireless. Communication through wired interface is very intricate and hard to implement or install. Wireless interfaces are chosen because they are easy to organize and install. Furthermore, Zigbee has some technical advantages over bluetooth, WiFi, infrared rays etc. Zigbee is a kind of low power-consuming communication technology for coverage area surrounded by $200 m$, with a data rate ranging from $20 Kbps$ to $250 Kbps$, it is appropriate for use in home area networks, mainly for the remote control of electric home appliances. Table. 1 shows the comparisons of bluetooth, WiFi, Zigbee.

```latex
\begin{tabular} { | p {1cm}| p {1.1cm}| p{1.2cm}| p {1.1cm}| p {1.2cm}| p{.6cm}|}
   \hline
    \textbf{Standard} & \textbf{Range} & \textbf{Number \newline of Nodes} & \textbf{Frequency \newline Band} & \textbf{Data \newline Protection }& \textbf{Power \newline use} \\ \hline
    \textbf{Bluetooth} & 10 m & 8 & 2.4GHz & 16 bit CRC & high  \\ \hline
    \textbf{Wi-fi} & 100 m & 32 & 3.1-10.6 GHz & 32 bit CRC & high \\ \hline
    \textbf{Zigbee }& 10-200 m & More than 25400 & 868/915 Mhz 2.4 GHz & 16 bit CRC & low  \\ \hline
    \end{tabular}
```

In this paper, we discuss different options of hardware technique for power controlling and monitoring architecture. For Monitoring, hardware is based on current or voltage measuring circuits, Micro Controller Unit (MCU) relay and Zigbee Reduce Function Device (RFD). Current/voltage measuring circuit measures the I/V and sends the information to MCU. Micro Controller checks abnormality of power and send the information to the home server where database is maintained through Zigbee RFD. For controlling purpose, relay is added in power monitoring hardware. In case of emergency found by MCU, relay cuts the power supply to the electric home appliances after receiving the control command. Graphic User Interface (GUI) software is used as an interface between user and end devices. User can control all electric appliances through cell phone, computer or laptop.

# Related Work and Motivation

For remote power control and monitoring, many wireless technologies are discussed; these are: Infrared rays, WLAN, Bluetooth, and Zigbee. However, Zigbee is suitable for remote power control and monitoring due to large coverage area up to $200 m$ and with transmission rate ranging from $20 Kbps$ to $250 Kbps$. Typically home appliance has three power modes, normal, stand by and off. In normal mode, home appliances use maximum power, however, in stand by mode, home appliances use almost $10$ percent of power consumption as compared to normal mode. In \[1\], authors discuss architecture of automatic cut-off for home appliances during stand by mode to save power consumption.

In \[2\], \[3\] and \[4\], power monitoring module calculates power consumption of electric home appliances in normal mode. Zigbee communication module is used to send measured data of current and voltage to server module and store it in computer. Embedded board is used instead of computer to reduce power consumption. In case of overload/abnormality in power consumption, server module sends a control message to MCU via zigbee modul. MCU cuts power to the load for safety purpose. Authors design the GUI by using Visual Basic (VB) to provide user friendly environment. However, power supply to power monitoring and controlling component is not discussed by authors.

In \[5\], Bluetooth is used for controlling electric home appliances. Bluetooth has low coverage area up to $10 m$. Bluetooth is better option for short distance controlling, however, for long distance controlling is not easy to achieve via bluetooth.

In \[6\], author discuss use of Zigbee communication in Advance Metering Interface (AMI). Zigbee is used to transmit detailed information of power consumption and also discuss the way of joining Zigbee network.

# Hardware

In this section, we discuss different hardware options with different techniques for power monitoring and control purpose in smart grid.

## Power Management Structure

In \[3\], power management structure for electric outlet as shown in Fig. 1, has three main parts: Zigbee based power monitoring module, home server and remote control section. Power monitoring module based on (MCU), Solid State Relay (SSR), a current measuring circuit and a Zigbee End Device (ZED). The home sever module has two parts: one is Zigbee Coordinator (ZC) and other is embedded board. The remote control section can be a personal Computer (PC) or Personal Digital Assistant (PDA).

Sensor in Zigbee based power monitoring module senses electric current being used by electric outlet. Current measuring circuit measures current and sends to MCU. MCU turns the electric outlet on/off. MCU based on two-way control mechanism: one direction for on/off control and other for current measurement. MCU handles ID of each end device, control electric outlets and send control message to home server through Zigbee coordinator. Through MCU, we can detect condition of all electric home appliances. Embedded system stores data of all power outlets and serve the user when they want.

To minimize power consumption, traditional relay is replaced by SSR for controlling on/off power outlets. In Zigbee based monitoring module, embedded board is used instead of PC to minimize the size, power consumption, and cost of home server.

## Real-time Power Monitoring Architecture With Direct Load Control

In \[4\], Zigbee based real-time power monitoring architecture consists of three modules: power management module, Remote Monitoring Control Module (RMCM) and user module. Power management module contains Power Observing Side (POS) and Power Control Side (PCS). POS monitors power consumption from various facilities like power consumption through switchboard and substation. Alternatively, large power utilization facilities such as central air conditioner, controlled by PCS. RMCM enclose major software components of POS and database. ZC is linked to RMCM through RS-232 interface for sending and receiving messages to Zigbee device on power management module. On user side, user can remotely operate POS through any web browser on notebook or PC, as shown in Fig. 2.

POS has three main parts Signal Transformation Circuit (STC), Digital Signal Processing (DSP) board and RFD. STC collects power consumption data from switch/substation and scale the information to fit input scope of Analog to Digital Converter (A/C) convertor of DSP board. DSP board is liable to calculate various power parameters and become aware of power omission in real time. This power information is transmitted to server side through Zigbee RFD. Zigbee RFD is an end device that can collect data through different sensors or switches and send to gateway (coordinator or router).

In power control, Zigbee RFD is directly connected to relay. After finding power abnormality ZC sends command to Zigbee RFD to unload assigned power facilities.

RMCS has following major functional components. Authentication and Authorization component (AA) verifying consumer identity and used for security purpose.Warning Message Transmission (WMT) component is capable of sending warning message to user through e-mails and mobile phone short message web service after finding abnormality in power. GUI component provides different user-friendly graphical web interface to control system through general browsers. Through remote control component, user can manually perform Direct Load Control (DLC). Data management component manages different data in system like user data, historical data of power consumption. The power demand forecasts component forecast the best possible contract capacity based on historical power consumption data.

## Power Observing Structure using Bluetooth, Ethernet and GSM

In \[5\], Bluetooth Controlled Power Outlet Module (BCPOM) is hardware architecture for power monitoring and controlling, as shown in Fig. 3. This hardware consists on several AC power sockets, vital control side, Protected Digital Card (PDC) side, Scalable Source Routing (SSR), power measuring side and communication interfaces based on Zigbee and GSM. SSR is used to turn on/off each sokets where electric home machines/appliances can be pluged.

Vital control side is based on four functions: *(i)* dealing out commands from communication interfaces bluetooth, ethernet and GSM, *(ii)* scheming the SSR on/off, *(iii)* observing the status of electric home appliances, and *(iv)* transmitting the power status and measured data to the protected Digital Card Side. GSM interface is used to connect vital control side and also allows to make a call by using GSM network. Bluetooth side is a low power embedded bluetooth with a built-in high-output antenna.

The ethernet side connects electric home machines to internet. PDC side is used for storing the measurement data of electric home machines and also store status of these appliances.

## Power Monitoring and Controlling Architecture

In \[2\], proposed system for power monitoring is based on three components: Data Acquisition (DA), Data Processing (DP) module and Application. Data acquisition consist of wireless sensors which are used for measuring AC and control power outlets. A wireless sensor has three technical parts, MCU with A/C, a sensing unit and a Zigbee transceiver. DP module collects information from all sensors through Zigbee and make a database of all collected information and responds to appeal from users.

## Power Controlling Architecture

Authors in \[1\] discuss the different techniques for controlling power in different conditions. One is automatic power cut off during standby mode. Architecture of automatic power cut off during standby condition of power outlet is based on A/C conversion, a two port relay, a micro controller and power observing circuit. AC input is connected to two port relay. One port of relay is directly connected to AC output electric power outlet and other relay is connected to the output relay through power observing circuit. Power observing circuit has three major components: transformer, rectifying diodes and additional components. In micro controller unit Zigbee RF module is added to communicate with the coordinator.

Other is Zigbee controller with IR code Learning Functionality. ZC is based on A/C conversion circuit, a MCU with Zigbee RF module, IR receiver and several switches. These several switches have different tasks. First switch is assigned to the first power outlet; through this button first power outlet will be controlled. Next two buttons can be assigned for the light control. One button is for 'dark' function and other is for 'light' function.

## Zigbee Based Transmission Line Monitoring

In \[7\], authors propose a Zigbee based transmission line monitoring system. The block diagram of the system is shown in Fig. 4. This system is based on Information Collection Unit (ICU), ZC and data management center. Information collection unit consists of sensor A/C, DSP, Zigbee RFD. Sensors keep information of power consumption in analog form and A/C convert analog information into digital information and send to DSP unit. DSP unit calculates variation in power consumption or any abnormality in power utilization and send the information to ZC through Zigbee RFD. ZC sends all information coming from different sensor to data management center.

# Suitability of IEEE802.15.4 for Smart Grid

There are various types of wired and wireless interfaces that can be used in smart grid for communication purpose similar to Bluetooth, WiFi, LAN, Zigbee, Ethernet, IEEE 1394, PLC, etc. Communication through wired interface in smart grid is very intricate and hard to implement and install; that is why wireless interfaces are chosen, since they are easy to organize and install. Furthermore, Zigbee has some technical advantages over others. It is low-cost, low power consuming wireless communication standard with maximum number of nodes. Zigbee has quicker response time. In \[6\], a new Zigbee based power meters can easily join the Zigbee network. Power meter sends the date and time synchronization command to the Zigbee device which sends this command to ZC. ZC replies with date and time data to the Zigbee device after completing the date time synchronization power meter ready to send the required data and become a part of Zigbee network. Below we tell the Open System Interconnection (OSI) model layer on which Zigbee protocol based and Zigbee based power monitoring and controlling.

## Zigbee

In \[8\], mound of Zigbee protocol is based on four layers of standard Open System Interconnection (OSI) model. Application Layer, Network Layer, Medium Access Control Layer (MAC) and Physical Layer. Application Sub layer layer is in charge for maintenance of required tables that match two devices according to their desires and services. Network layer is responsible for creating network by adding and deleting the nodes and for the route discovery between devices and their maintenance. Zigbee has two standards for physical layers that work in different frequency range: 868 MHz, 915 MHz and 2.4 GHz. MAC layer is conscientious to offer interface between the Service Specific Convergence Sub layer (SSCS) and physical layer.

## Zigbee Based Monitoring and Controlling

Zigbee technology can be implemented in generation, transmission, distribution, and in consumer sectors to provide control on power consumption and give accurate data to user and utility. Fig. 5 shows communication architecture, where generation side, transmission side, distribution side and consumers are not only connected electrically but also connected by communication interfaces with markets, operations and service providers. Zigbee based devices have two types: one is Full Function Device (FFD) and other is RFD. FFD can be coordinator or router. It is liable to create the network, select the radio frequency channel and distinctive network identifier to avoid collision among data from different sensor nodes and for security purpose. Zigbee RFD plays role of end device that collects variety of data from different switches and sensors. Here we discuss Zigbee based transmission lines monitoring, meter reading system and load control system.

### Zigbee Based Transmission Lines Monitoring

In \[7\], transmission line monitoring system is based on environmental parameters of lines and towers such as temperature, ice, wind lightening etc. Sensors are placed on transmission line. These Zigbee based sensors sense power passing through it and send to data center via Zigbee gateway that are fixed on transmission tower. Small blue boxes represent power monitoring device as shown in Fig. 6.

### Zigbee Based Meter Reading System

In \[8\], Zigbee based meter reading system has two main parts: Zigbee network and database unit. Zigbee network is constructed by Zigbee router, end devices and coordinator. Router and coordinator must be FFDs and end devices can be RFDs. After calculating required information end devices send this information to the relevant router. Router has function to synchronize service to other FFDs like coordinator. Coordinators send received data through $RS232$ interface to database unit.

## Zigbee Based Load Control System

In \[1,4\], for controlling purpose, Zigbee RFD is directly connected to relay. Relay is used for turning the switch on and off. After finding any abnormality in power used by electric home appliances control command is sent to the Zigbee RFD by coordinator. Subsequently, relay turns off those home appliances. ZC also receives the data from the end devices through the zigbee RFD.

# User responsive control system

Zigbee based Smart meters are part of AMI which give all information about power to customer like showing peak hours time, current running price, and real time power consumption by electric home appliances. It involves user to control power load at his end. This system provides reliable control system to user through different ways.

## Control by Internet

In \[5\], the software for remote controlling is install in PC or laptop and connect to internet. The network formation of connecting remote control by internet is a host-client structure. This remote control software sets IP address and ports and sends request to Power Monitoring System (PMS), creates a link between user and PMS. In \[4\], when PMS detects a power abnormality, warning message hurl to user through e-mail by means of Mail Transfer Protocol (MTP). In \[2\], under the proposed system, functional schedules are implemented on java Virtual Machine(VM). When user want to switch on/off an electric home appliance from a web interface, response time is less than one sec.

## Control by PDA with GUI

In \[1\], \[3\], \[4\] and \[5\], PDA is connected to power monitoring module for controlling and monitoring electric home machines. GUI is used in control area to create interface between user and electronic devices. By making use of GUI, user can access condition of electric home appliances, time reaction and energy utilization made by electric power outlets. GUI provides effortless control of power status of electric home machines. User can set each switch as on/off by sending command. The remote control system offers to supervise and manage power condition of electric home appliances.

## Control by GSM Cellular Mobile Phone

In \[5\], consumer can examine and organize electric home machines using GSM cellular mobile phone. User sends control command message through GSM cellular mobile phone to PMS. After receiving control command PMS allows user to control if sending command format match with system. User can be capable for supervise and organize power condition of electric home appliances anytime and anywhere using GSM cellular mobile. PMS can also send warning message to consumer after finding unusual status of power

# Conclusion

In this paper, we discussed power utilization, power organizing and power controlling architecture for power saving purpose. We also discussed the role of Zigbee in transmission line monitoring, real time meter reading and direct load controlling of electric home appliances. This paper also describes the user friendly control home appliances for power on/off through internet, PDA using GUI and through GSM cellular mobile phone.


abstract: Each year, thousands of software vulnerabilities are discovered and reported to the public. Unpatched known vulnerabilities are a significant security risk. It is imperative that software vendors quickly provide patches once vulnerabilities are known and users quickly install those patches as soon as they are available. However, most vulnerabilities are never actually exploited. Since writing, testing, and installing software patches can involve considerable resources, it would be desirable to prioritize the remediation of vulnerabilities that are likely to be exploited. Several published research studies have reported moderate success in applying machine learning techniques to the task of predicting whether a vulnerability will be exploited. These approaches typically use features derived from vulnerability databases (such as the summary text describing the vulnerability) or social media posts that mention the vulnerability by name. However, these prior studies share multiple methodological shortcomings that inflate predictive power of these approaches. We replicate key portions of the prior work, compare their approaches, and show how selection of training and test data critically affect the estimated performance of predictive models. The results of this study point to important methodological considerations that should be taken into account so that results reflect real-world utility.
author: Benjamin L. Bullough

Anna K. Yanchenko

Christopher L. Smith

; Joseph R. Zipkin

bibliography: iwspa01-bullough.bib
title: Predicting Exploitation of Disclosed Software Vulnerabilities Using Open-source Data

# Introduction

Thousands of software vulnerabilities are discovered and reported to the public every year. Once a vulnerability becomes public, the odds of it being exploited increase drastically , making unpatched disclosed vulnerabilities a significant security risk. Software vendors are generally quick to provide patches once vulnerabilities are known, and vigilant users are quick to install those patches as soon as they are available. However, writing, testing, and installing patches can involve considerable resources, requiring organizations to prioritize vulnerabilities based on some notion of risk. Because most software vulnerabilities are never actually exploited , it would be desirable to consider the probability that they will be exploited as part of the risk assessment process.

Several previous studies have characterized the exploitation of disclosed vulnerabilities. Bilge and Dumitras found that "after vulnerabilities are disclosed publicly, the volume of attacks exploiting them increases by 5 orders of magnitude". They also found that "exploits for 42% of all vulnerabilities employed in host-based threats are detected in field data within 30 days after the disclosure date." This result illustrates the risk posed by unpatched software vulnerabilities, the need for software vendors and users to quickly provide and install patches and the impact of a failure to patch.

However, Nayak et. al. find that only 15% of the known vulnerabilities in a set of popular software products (i.e. MS Windows, MS Office, Internet Explorer, Adobe Reader) are ever exploited "in the wild" (i.e. among computers used by the general public ). They also find that a small subset of vulnerabilities contribute disproportionately to attacks. This finding suggests that not all vulnerabilities carry the same risk and thus software makers and system administrators could benefit from a method for prioritizing their response to disclosed software vulnerabilities.

Frei et al. study the lifecycle of vulnerabilities including creation, discovery, disclosure, exploit availability, patch availability and patch installation. They describe the major players in a "security ecosystem" and how the motives of the different players result in different orderings of the milestones in the vulnerability lifecycle. Their empirical results show that in most years over 70% of all exploited vulnerabilities have exploits available at the time of disclosure.

While there are many sources of data on software vulnerabilities a few deserve special mention due to their ubiquity in research and practice. The Common Vulnerabilities and Exposures (CVE) dictionary, maintained by the MITRE Corporation, is an authoritative source of identifiers for publicly known information security vulnerabilities . The National Vulnerability Database (NVD) is a "repository of standards based vulnerability management data" produced by the U.S. National Institute of Standards and Technology (NIST) . The NVD is a superset of the CVE dictionary and is automatically synchronized with it. The Common Vulnerability Scoring System (CVSS) is an "open framework for communicating the characteristics and impacts of IT vulnerabilities" . While CVSS has several components, the CVSS Base Score is the most widely used. The CVSS Base Score is given on scale from 0 to 10, calculated from a vector of categorical variables that reflect the characteristics of the vulnerability.

For lack of a better method, the CVSS Base Score is commonly used as an indicator of risk and this practices has been codified in some industry security standards . However, Allodi and Massacci show that the CVSS base score alone is a very poor indicator of whether a vulnerability is likely to be exploited in the wild . They find that the existence of a proof-of-concept exploit and the existence of an exploit traded in cybercrime black markets are much more indicative of the risk of exploitation.

# Related Work

Several studies have attempted predict with machine learning techniques whether disclosed software vulnerability will be exploited. Bozorgi et. al. use features derived from the Open Source Vulnerability Database (OSVDB) and CVE to train a support vector machine (SVM) classifier. As ground truth they use the OSVDB "Exploit Classification" status, coding as positive any entries listed as "available, rumored or private exploit". Their dataset contains vulnerabilities disclosed from 1991–2007. They report classification accuracy approaching 90% in offline experiments using a resampled, balanced data set. They also experiment with online exploit prediction, where classifiers are repeatedly trained with all vulnerabilities seen up to a point in time, and find that classification accuracy stabilizes near 85%. Interestingly, 73% of the vulnerabilities in their data set are labeled as exploited (N=13,765), which contrasts with other surveys and datasets that show most vulnerabilities are not exploited.

Edkrantz and Said apply several machine learning algorithms to the task of classifying which vulnerabilities will be exploited. They use vulnerability records from the NVD as a source of features. They use the presence of an exploit in Exploit Database (EDB) that references a CVE-ID as a source of ground truth for their study. (Exploit Database is an archive of software exploits maintained by the company Offensive Security.) In their overall dataset, which covers the years 2005 through 2014, 27% of vulnerabilities are labeled as positive (N=55,914). They report 83% accuracy on a balanced test set using SVM with a linear kernel (with slightly worse results using Random Forest and Naı̈ve Bayes classifiers).

Building on the work of Bozorgi , Sabottke et al. use linear SVM classifiers to predict which vulnerabilities will be exploited, but instead of using features generated from the vulnerability description text in the NVD database, they substitute the text of Twitter posts (tweets) that reference a particular CVE-ID. In addition to predicting public "proof-of-concept" exploits that are found in databases such as EDB, they consider prediction of "real-world" exploits, based on Symantec anti-virus and intrusion detection signatures, and private proof-of-concept exploits, based on Microsoft's Exploitability Index. While their data is restricted to the one-year period from January 2014 to January 2015, it is notable that the class ratios are starkly different than in Bozorgi. While the majority of vulnerabilities in the Bozorgi data are labeled as exploited, only 6.2% of vulnerabilities in Sabottke's dataset have corresponding private proof-of-concept exploits and only 1.3% have real-world exploits (N=5865).

# Challenges of Realistic Exploit Prediction and System Evaluation

Machine learning techniques for classification present a compelling approach for the task of predicting which software vulnerabilities will be exploited due to their ability to discover complex patterns within high-dimensional data. However, care must be taken in selecting training and test data and generating features so that the results will be representative of performance when used for online prediction to aid enterprise network defenders. We consider several issues that critically affect the evaluation of predictive models and offer suggestions for evaluation that will more readily translate to real-world performance. These challenges are realistic class balance, appropriate seperation of training and test data, representative and realistic test data and meaningful benchmarks.

In their work surveying the field of intrusion detection, Sommer and Paxson examine why machine learning approaches have not been widely deployed despite extensive academic research. They describe a number of challenges that make this domain particularly difficult, including the difficulty of evaluation, and offer guidelines that they hope will improve the utility of research. Exploit prediction involves many of the same challenges and adds the additional difficulty of forecasting future events rather than detecting ongoing attacks. In a similar vein, Rossow et al. describe many of the pitfalls that can limit the correctness and realism of experiments using malware . While the context is different from that of exploit prediction, several of the issues they discuss echo findings in this work, including the appropriate seperation of training and evaluation data, removing moot samples, the appropriate balance of classes and use of real-world data.

```latex
\begin{table*}[t]\centering
\renewcommand{\arraystretch}{2}
\begin{tabular}{ | l | c | p{13cm} |}
\hline
Metric & Formula & Description \\ \hline
Accuracy & $ \frac{TP + TN}{TP + TN + FP + FN} $ & Fraction of predictions that are correct \\ \hline
Precision & $ \frac{TP}{TP + FP} $ & Fraction of positive predictions that are correct \\ \hline
Recall & $ \frac{TP}{TP + FN} $  & Fraction of cases with the positive condition that are correctly predicted \\ \hline
F1 & $ 2 * \frac{precision * recall}{precision + recall}  $ & Harmonic mean of Precision and Recall \\ \hline
\end{tabular}
\captionof{table}{Classification performance metrics.  The following abbreviations are used: TP - true positives, TN - true negatives, FP - false positives, FN - false negatives}\label{tab:metrics}
\end{table*}
```

A variety of metrics are commonly used to evaluate the performance of a classifier, such as accuracy, precision, recall and F1 score. These are summarized in . Accuracy is the fraction of predictions that are correct. While it is good to have high accuracy, its drawback as a performance metric is that it fails to distinguish between different types of prediction errors. Conflating the different types of error can be misleading when their costs are different or when the class ratios are highly imbalanced. For example, when the classes are highly imbalanced, it becomes increasingly trivial to achieve high prediction accuracy by simply predicting the majority class in every case.

Using precision and recall together distinguishes between the impact of errors that are false positives from those that are false negatives. Precision is defined as the fraction of positive predictions that are correct and highlights the effect of false positives. Recall is defined as the fraction of cases with the positive condition that are correctly predicted and highlights the effect of false negatives.

The F1 metric is the harmonic mean of precision and recall and provides a single number that summarizes the performance of a classifier. As there is an inherent trade-off between precision and recall, most classifiers can be tuned to select an operating point that is optimized for a particular application. Precision-recall curves (e.g. ) summarize graphically this trade-off for the range of possible values of recall.

## Class Imbalance

The ratio of class labels is an important consideration in training and evaluating a classifier. It is generally more difficult to build a classifier that performs well on both precision and recall when the class ratio is highly imbalanced. Performance measured on an artificially balanced dataset will generally not be representative of performance when the classes are imbalanced. Therefore, it is essential to evaluate a classifier on a test set that is representative of the true class ratios; otherwise, the results will be misleading.

## Train/Test Split

A common practice when developing machine learning models is to split a labeled dataset into two parts, one for training the model and another that is reserved exclusively for evaluating the performance of the model. A typical method is to partition the data randomly. This is a best practice when the samples can be assumed to be mutually independent and drawn from the same probability distribution; however, this assumption is not strictly true in the case of exploit prediction. While some violations of this assumption may be acceptable, it can be problematic in the case of predictive analytics to train models based on information that is available only in the future. For example, if a new term is coined in the future to describe something about the positive class, it is not reasonable to assume that a model could learn a feature that accounts for its presence based on past training data. Temporal intermixing of training and test data violates this common sense assumption, and allows information about the future to "leak" into the model. While this could be rationalized as simply ignoring the effects of concept drift, accounting for the impact of concept drift is a critical aspect of evaluating any candidate predictive model.

## Representative and Realistic Data

In addition to the issues mentioned above, it is important that the data represent the task faithfully in other respects. In the case of exploit prediction, the goal is to predict whether a vulnerability will be exploited in the future. However, many vulnerabilities are known to be exploited at the time of disclosure. Predicting whether such vulnerabilities will be exploited is not a meaningful task and these cases should not be included as test samples. Similarly, it is important to consider whether a realistic system would be operating in a batch mode, where many historical examples are considered at one time, or an online mode, where each sample is classified as it arrives. Some features that can be calculated retrospectively in a batch setting use information that might not be available in an online setting, which is more representative of real predictive analytics applications. For example, if vulnerabilities were to be classified at the time of disclosure, information about the elapsed time between record creation and any subsequent record modifications would not be available.

## Meaningful Benchmarks

Reporting that a particular technique achieves a certain level of performance on a representative task is more useful when given in the context of the next best alternative. Only by doing so can actual progress be measured. Comparing an approach to a naı̈ve model only shows that the approach is not naı̈ve, not that it advances the state of the art in a meaningful way

## Examples from Related Work

Much of the prior work in exploit prediction takes an approach at odds with one or more the aforementioned principles. Bozorgi and Edkrantz each evaluate their methods using artificially balanced datasets, inflating performance metrics. Edkrantz and Sabottke each randomly split training and test data, allowing the temporal intermixing of "future" and "past" data in the training and test sets. Many of the most influential features in the model used by Bozorgi are derived from information that would not be available in an online setting (though they do present results from a simulated online experiment). Sabottke evaluates the utility of text-based features derived from social media, but never compares those results to using text features derived from the vulnerability summary that is readily available at the time of disclosure.

# Method

In this study, we follow the outlines of prior work to build a model to predict which software vulnerabilities will be exploited. However, our focus is on illustrating the impact of the previously discussed challenges. To this end, we systematically vary the approach of selecting training and test examples and generating features to highlight the effects on prediction performance.

While our approach to training the baseline models follows many of the common aspects of previous work, we are unable to precisely duplicate any particular study due to the unavailability of data or a lack of detailed knowledge about the methods used. (For example, OSVDB dataset is no longer publicly accessible.) However, we find that our baseline approach achieves results similar enough to be comparable, and we focus our attention on evaluating the effects of controlled changes to the model building and evaluation methodology.

Name
Type
Count

CVSS
Access Vector
C
3

Access
C
3

Authentication
C
3

Confidentiality Impact
C
3

Integrity Impact
C
3

Availability
C
3

Score
N
1

1-4

Hardware Count
N
1

2-4
OS Count
N
1

2-4
Applications Count
N
1

2-4
List
T
2000

1-4

ID
C
3

2-4
Name
T
51

2-4
Description
T
178

2-4

N
1

2-4
Parents
C
30

2-4
Parents' Names
T
29

2-4
Parents'
T
108

1-4

Number
N
1

2-4
Type
T
3

2-4
Source
T
42

2-4
URL
T
2000

1-4 Summary
Summary
T
2000

1-4

We obtained data about published vulnerabilities from the NVD, which indexes vulnerabilities by their CVE-ID . Our dataset contained 38,129 vulnerabilities published between 2009 and 2015. shows the distribution of the disclosure dates over time. Each record includes a descriptive text summary for the vulnerability, scores and metrics from CVSS, information about affected products and vendors, the category of vulnerability based on the Common Weakness Enumeration (CWE) system, and URLs to other reference sources. contains a list of all data features and their type (i.e. numeric, categorical or text).

The numeric CVSS score vulnerability is calculated from the values assigned to the feature's Access Vector, Access Complexity, Authentication, Confidentiality Impact, Integrity Impact, and Availability Impact. Access Vector can take on the values "requires local access", "adjacent network accessible" or "network accessible". Access Complexity can be rated "high", "medium" or "low". Authentication can take on the values "requires multiple instances of authentication", "requires single instance of authentication" or "requires no authentication". Confidentiality Impact, Integrity Impact, and Availability Impact can be scored as "none", "partial" or "complete". These numbers are used to calculate the overall, numeric CVSS score (see ).

Text features that had multiple entries for the same CVE-ID, (i.e. the Vulnerable Systems List, Reference Types, Reference Sources, Reference URLs, CWE Names of Parents, and CWE Descriptions of Parents) were concatenated together into one longer entry for that specific feature. The counts of hardware, operating systems, and applications affected were aggregated from the Vulnerable Systems List. Each CWE has a name and description and there is a hierarchy of CWEs, with CWE-IDs lower on the tree more specific than those above.

We merged the NVD data with data from the Exploit Database by CVE-ID. This data was then further pre-processed, according to feature type. Numerical features were standardized to have zero mean and unit variance, categorical features were represented using binary one-hot encodings, and text features were represented using a bag-of-words model. Each group of text features was individually modeled using a separate term frequency-inverse document frequency (TF-IDF) model, with a maximum of 2000 terms. (We found there was minimal benefit in using more than 2000 terms.) Stop words were removed from all text features and the terms "Exploit-DB", "Milw0rm", and "OSVDB" were removed from the Reference Source. Reference URLs that contained any of these three terms were completely removed. These three sources are all databases detailing exploits and they were excluded so as to not leak information about already known exploits into the model. While the presence of these terms in the domain would be an obvious indicator that the vulnerability had been exploited, other tokens associated with the URL structure for each site could be equally telling, thus we remove the entire URL before extracting features.

For our ground truth labels, we used the presence of a proof-of-concept (POC) exploit in the Exploit Database (EDB) to choose whether to mark a CVE as exploited or not. CVE references to EDB are collected in a reference map at (2319 unique CVEs link to EDB); however, many more EDB entries contain links to CVE-IDs that are missing from this list. To compile a complete list, we used a web scraper to check every EDB entry for a CVE link to generate a full EDB–CVE mapping. This full mapping gave us a list of 19,864 unique CVE entries with POC exploits present in the EDB. Of these, 6470 were within our target date range.

After splitting our data into training and test sets and pre-processing our features, we trained a support vector machine (SVM) classifier with a linear kernel on the training data. We used the trained classifier to generate predictions on the test set. Rather than make a hard classification decision for each test sample and produce single point estimates of precision, recall, or accuracy, we use the raw output of the decision function to plot interpolated precision and recall plots, which show the entire range of precision-recall performance that would be achievable with the classifier depending on the selected threshold.

| Name                        | Type | Count |
|:----------------------------|-----:|------:|
| \# tweets                   |    N |     1 |
| \# with high friend count   |    N |     1 |
| \# with high follower count |    N |     1 |
| \# retweets                 |    N |     1 |
| \# favorited                |    N |     1 |
| Ave \# hashtags             |    N |     1 |
| Ave \# URLs                 |    N |     1 |
| Ave \# mentions             |    N |     1 |
| \# verified                 |    N |     1 |
| Ave account Age             |    N |     1 |
| Body                        |    T |  2000 |

For our experiments with Twitter data we collected tweets from the Twitter API that contained the term "CVE" and selected those tweets containing a valid CVE-ID. Our Twitter collection covered the first 5 months of 2016. We then considered only those CVE records for which there was at least one tweet in this time period (2712 in total). Because our collection system only collects tweets containing the query term "CVE", and these can be expected to account for much less than 1% of all Twitter traffic, we can reasonably assume that our corpus contains all of the matching tweets over the collection time period, despite the 1% rate limit imposed by the Twitter API . Our approach is very similar to Sabottke. Although our data covers a different time period, we find that the number of unique CVE-IDs in our corpus relative to the collection duration is proportional to that of Sabottke.

describes the features, derived from , that were used in these experiments in addition to the CVSS features from . The statistical features are calculated from the collection of tweets that reference a particular CVE-ID. The lexical features are based on the concatenated text from the bodies of the tweets. As before we use the presence of the CVE-ID in EDB to assign the class label.

# Results

## Experiment 1: Class Imbalance

To see the effects of class imbalance, we manipulated the class ratio of our datasets by resampling from our original data set, of which 17% of the vulnerabilities were exploited (positive class). To make our results comparable to prior work, we randomly split the data between training and test sets, where 18% of the data was in the test set. The results in and show that performance was significantly boosted by artificially generating balanced datasets. Likewise, performance dropped further if the imbalance became more severe, which may be realistic, as we discuss in . When the classes were balanced, performance exceeds the results reported by Edkrantz and approaches the results reported by Bozorgi .

## Experiment 2: Randomized vs. Temporal Train/Test Split

To see the effect of temporal intermixing of the training and test data, we compared splitting the data by date (where all vulnerabilities disclosed after January 1, 2015, were placed in the test set), to randomly selecting an equivalent portion of the data as the training set. Since all vulnerabilities published after January 1, 2015, accounted for about 18% of the data, we also used 18% of the data in the test set for the random split approach. We additionally evaluated an online approach where we trained on the first year of data and evaluated on the subsequent six months of data. We then added the test data to the training set, retrained the model, and evaluated it on the following six months, and so on. and demonstrate that random splitting of the training and test data substantially inflate the performance of the predictive model. The results using a sliding-window online approach are between the batch approaches, suggesting it is a reasonable approach that minimizes the negative effects of concept drift while properly restricting training data to events that have occurred in the past.

We note that the fraction of positive cases in the temporally split test set is only 9.3%, which is less than the 16.7% of positive cases in the randomly split test set. This shows that the class ratio in our data is not stable over time, and it means that some of the decrease in performance is attributable to the class imbalance being somewhat more extreme when we use only the more recent data for evaluation. However, given the quantitative results from our experiment with class imbalance, this amount of change in the class ratio is not enough to account for the majority of the effect on performance. To further examine the difference in performance between random and temporal splits, we looked at the class separation achieved by the classifier decision function learned with each type of split. shows significantly more class separation in the random split case, as compared to the temporal split case. The classifier was not able to distinguish between the positive and negative classes in the temporal spilt case as successfully as it was for the random split, leading to the difference in performance.

## Experiment 3: No Pre-exploited Vulnerabilities

It is important to measure the performance of a predictive model on a task that is representative of the realistic situation. In many cases, exploits are published before or concurrently with a vulnerability disclosure. shows a histogram of the difference in days between vulnerability disclosure in the NVD database and exploits that target those vulnerabilities being published in EDB. Most vulnerabilities in the NVD that are exploited have already been exploited by the time they are disclosed. Since a simple database search can determine that these have been exploited, they are not really candidates for prediction. When we excluded these examples from the dataset we found that performance was drastically reduced to a level not much better than guessing. Since this also exacerbates the class imbalance, we compared this performance to a dataset where a random sample of the positive cases was removed so that the class ratio was equivalent to the former case. We see in and that while the effect of increasing the class imbalance is substantial, performance is still much better than when all of the previously exploited cases are removed.

Since the task of predicting the exploitation of disclosed vulnerabilities is premised on the idea that exploits are commonly developed based on the vulnerability disclosures ("zero-day" exploits being the obvious exception), it was surprising to find such a large fraction of cases where the vulnerability publication date is later than the earliest publication date of a proof-of-concept exploit that references that vulnerability. describes how it is possible for an exploit to reference a vulnerability by its CVE-ID prior to its publication and provides examples of specific CVEs. While an exploit does exists at the time of vulnerability disclosure in many cases, a limitation of our data is that it does not indicate when any database updates that link the two may have occurred, which would better indicate public knowledge about the exploited status of a vulnerability. If such updates do not always occur very soon after the vulnerability publication, then our approach of removing all vulnerabilities from the dataset where the exploit is published prior to disclosure might seem overly aggressive. On the other hand, if we assume that any vulnerabilities can be quickly matched to existing exploits, then using a simple database lookup on past exploits as a classifier would achieve 100% precision at 59.3% recall on the full test set while ignoring any future exploit development. This suggests that any added value must come by improving performance on the case where exploits are not yet developed.

## Experiment 4: Twitter Text vs. NVD Summary Text

The lexical features derived from the NVD summary text account for almost all of our classifier's performance. and show that all the other features combined provide a very modest improvement, mainly when precision is emphasized over recall.

Sabottke investigates using features derived from Twitter to predict whether software vulnerabilities will be exploited. They combine lexical features from the text of tweets mentioning CVEs with other statistical features from the Twitter data and features from the NVD database. While using social media appears to produce promising results, they do not compare their results to a baseline approach of using the lexical features in the NVD summary text. compares our results using features derived from the NVD database (summary and CVSS scores) to our results using features derived from Twitter along with the CVSS scores. (The results using NVD text in this experiment differ from due to using a different range of time.) We show the results using only the lexical features as well as augmented with most of the statistical features used by Sabottke. In both cases, the model using Twitter features does not show an improvement over the baseline model.

Collecting information about vulnerabilities from Twitter involves a significant investment in data processing capabilities, delays the time to make a prediction until the Twitter community generates sufficient tweets, and can only be applied to vulnerabilities that generate social media interest. The open nature of the Twitter platform also poses a greater risk of an adversary attempting to poison the defenders training data, though this was investigated by Sabottke who found the impact would be modest under several models of adversarial interference. Given these considerations the marginal benefits of using social media would need to be sufficient to justify the costs. Our results suggest that such a benefit does not exist.

# Discussion

The results of the four experiments described in the previous section show that subtle changes in the evaluation process can have a profound impact on the estimated performance of a predictive system. Making more realistic choices about class balance, how to split training and test data and what constitues a realistic example for prediction lowered the F1 score of our classifier from 83% to 4%. While the appropriate choices for each of these dimensions is highly dependent on the prediction task and training data, this example illustrates why it is important to make realistic decisions for each of them in order to get performance estimates that are likely to translate to a pratical setting.

Our assessment of the value of predictions based on NVD data is consistent with the results of Zhang et al. , who attempt to use NVD data to predict the time until the next *undiscovered* vulnerability is reported in a particular piece of software. They conclude that data in the NVD generally has poor predictive capability. One reason that NVD data may have poor predictive power is the presence of substantial concept drift (as seen in ), which makes it difficult to generalize about future vulnerabilities based on descriptions of past vulnerabilities.

We also find limited use for social media in predicting whether software vulnerabilities will be exploited. While we focus on the issue of performance relative to an established baseline, Tufekci describes several methodological pitfalls, such as over-reliance on a single platform, unrepresentative samples, and field effects that should also be considered when using social media big data for predictive analytics.

While models based on the open source database information examined in this study led to poor prediction performance, more accurate predictions might be possible by exploiting better sources of data. Developing richer or higher-fidelity databases about vulnerabilities and exploits, perhaps by collaboration with organizations that hold large amounts of private data, is one approach that exploits the potential of big data for security analytics. Another avenue might be to use static or dynamic analysis of vulnerable code or binaries to obtain greater insight into the fundamental exploitability of a particular vulnerability or its utility as a target for exploitation. Exploring such approaches is left for future work.

A limitation on effectiveness of any predictive system operating in an adversarial environment is that its predictions may influence future events, thereby undermining the apparent accuracy of its predictions. For example, if predictions about software exploitation become widely known and relied upon, attackers may intentionally shift their focus to another less defended target. Such an offensive strategy presumably imposes some cost on attackers by pushing them towards less desirable avenues of attack, so there is still some net benefit to the defender for deploying the predictive system. This limitation exists regardless of the mechanism used for prediction, but systems based on machine learning may be quicker to adapt to evolving attacker strategy than systems based on manually developed rules. Exploring the co-evolution of attacker and defender strategies and the impact on the effectiveness of defensive systems is also left to future work.

# Conclusions

In this study we identify several methodological considerations that need to be accounted for during training and evaluation of predictive models that will be used in practical settings. Previous studies that have investigated predicting exploits of disclosed software vulnerabilities have failed to account for some or all of these issues. We show empirically the impact of violating these principles on the task of exploit prediction. We conclude that models of software vulnerability exploitation based on NVD database entries and social media alone are unlikely to have enough predictive power to be useful in practice.

# Acknowledgments

We thank Jeremy Blackthorne and Clark Wood for their helpful feedback on drafts of this paper.

# Vulnerabilities with Pre-existing Exploits

The median difference in time between exploit publication and vulnerability publication in our dataset is 8 days, but there are many individual cases where this gap is much larger. One might ask how it is possible to reference a vulnerability by its CVE-ID prior to its publication. One explanation is that CVE-IDs can be allocated prior to publication in the NVD database. This would allow a security researcher who finds a vulnerability to reserve a CVE-ID, publish their proof of concept exploit and then publish the vulnerability, in that order. (The researcher may also disclose the vulnerability privately to the vendor prior to public disclosure in accordance with responsible disclosure practices as described by Frei. ) An example that seems to fit this pattern is CVE-2015-4414, a directory-traversal vulnerability in a WordPress audio plugin that allows an attacker to read arbitrary files on the system. The CVE-ID for this vulnerability was reserved on 7 June 2015. An exploit (EDB-ID: 37274) that references this CVE-ID was published on EDB on 12 June 2015. The vulnerability was published on the NVD on 17 June 2015.

The record of an exploit might also be updated after its initial publication to reference a vulnerability that is subsequently published. This may be because the exploit author did not disclose the vulnerability and thus didn't know the CVE-ID prior to the vulnerability publication. It may be that a newly discovered vulnerability is readily exploited using an existing exploit with only a trivial modification. An example of the later case is CVE-2011-5289, published in the NVD on 31 December 2014, which exploits the SaveDecrypted method in the aTube Catcher ActiveX control to write arbitrary files. This vulnerability is referenced in EDB by an exploit (EDB-ID: 6963), which was published on 3 November 2008. This exploit also references an earlier vulnerability (CVE-2008-5002), published on 10 November 2008. This vulnerability affects the same ActiveX control and allows an attacker to overwrite arbitrary files via the WriteFile method.


abstract: The increased use of electronic health records has made possible the automated extraction of medical policies from patient records to aid in the development of clinical decision support systems. We adapted a boosted Statistical Relational Learning (SRL) framework to learn probabilistic rules from clinical hospital records for the management of physiologic parameters of children with severe cardiac or respiratory failure who were managed with extracorporeal membrane oxygenation. In this preliminary study, the results were promising. In particular, the algorithm returned logic rules for medical actions that are consistent with medical reasoning.
author: Michael A. Skinner,^1,2^ Lakshmi Raman, ^2^ Neel Shah, ^2^ Abdelaziz Farhat, ^2^Sriraam Natarajan, ^1^
^1^University of Texas at Dallas, Dallas TX
^2^University of Texas Southwestern Medical Center, Dallas TX
bibliography: ECMO_AAAI.bib
title: A Preliminary Approach for Learning Relational Policies for the Management of Critically Ill Children

# Introduction

The ability to automatically learn physician actions from electronic health records (EHR) could contribute to improved health care in a number of ways. For example, we could automatically discover optimal policies [^1] for managing particular diseases. Moreover, an optimal policy, once discovered, could be compared to a patient's actual clinical course; if there is a deviation, physicians could be provided with suggestions for care. Finally, the ability to extract medical polices from EHRs would enable predictions of patient prognosis and outcomes.

In this work-in-progress, we investigate use of a boosted SRL framework to elicit weighted first-order logic clauses mapping the values of a set of physiologic parameters to physician actions in critically ill patients with respiratory or cardiac failure. We extracted the information from clinical trajectories documented in the EHR. The goal of this work is to explore the use of such frameworks in this challenging medical task. [^2]

**The clinical challenge – discovering a medical policy** Unwanted variation in medical care, recognized for over forty years, remains a challenge to health care providers in nearly every specialty . Differences in care are typically observed between geographic regions, and the particular practice in an area often correlates with available resources. For example in one study, investigators found a high correlation between the availability of cardiac catheterization within a locality and use of angioplasty for managing cardiac disease . It is surprising that these challenges persist, even as there has been a multiplication of published expert guideline documents for many medical conditions whose recommendations are based on well-performed prospective clinical trials .

To decrease variability of care, and to converge medical management around policies conforming to expert guidelines, clinical decision support systems (CDS) have been devised to render advice to clinicians as they care for patients . Such systems were initially very limited, highly dependent on manual curation, and their scope was limited to a very few medical conditions . However, the increased use of EHRs has stimulated the development of automated CDS systems holding promise for providing advice to health care providers in real time.

Required in an automated CDS system is the ability to monitor some aspects of the patient's clinical state, as well as the physician actions . Moreover, the system must possess some notion of optimal care; when clinicians deviate from the preferred management, or if unexpected events occur warranting a change in care, alerts or reminders are provided from the system. Whereas early systems used hard-coded rules to encode clinicians' knowledge about the optimal policy, there is growing interest in automatically extracting optimal care patterns by mining the EHR .

Reinforcement learning (RL) is the most commonly reported technique to extract clinical policies from medical records . Somewhat surprisingly, to our knowledge, other policy learning schemes such as imitation learning have not been reported in the medical realm. Most of the reported RL models use deep neural networks, requiring many patient records for training and a propositionalization/embedding technique that could lead to loss of information. Moreover, these models may be difficult to interpret, complicating the identification of best medical actions for a given patient state.

A key issue when learning medical policies is that it is non-trivial to construct a vector-based representation for EHR information. There can be multiple measurements performed over varying time-scales, multiple treatments of different conditions at the same time-step and differing numbers of observations per subject. Thus, if standard machine learning methods are used to represent these complicated data, critical information may be lost.

A more natural representation that allows for modeling relational medical data is to use first-order logic. Hence, motivated by the fact that physicians generally make treatment decisions based on physical, laboratory and radiologic findings in a systematic manner through a series of (often implicit) "if-then" decisions, we investigated the usefulness of learning medical polices as sets of probabilistic clauses learned in a statistical relational learning framework .

# Problem Description - ECMO patients

Extracorporeal membrane oxygenation (ECMO) is a method of supporting patients with severe respiratory or cardiac failure. The technique requires placement of large cannulas in the neck or in the heart, and externally circulating the patient's blood through a system that oxygenates the blood and removes carbon dioxide. Reserved for the most critically ill of patients, mortality can be very high and even among survivors there are frequent treatment complications .

This study used de-identified medical data abstracted from EHRs for 140 children treated at the Children's Medical Center of Dallas who survived their period of ECMO. The study was performed in accordance with an exemption granted by the University of Texas Southwestern Institutional Review Board (IRB). The time on ECMO ranged from 6 to 985 hours, averaging 174 hours. For each hour of ECMO bypass, and for from 1 to 24 hours prior to cannulation (15 hours, on average), 40 physiologic and laboratory parameters were recorded. Not every parameter was measured each hour; for example, those exclusively associated with ongoing bypass (such as pump flow) were only recorded while the child was actually undergoing ECMO support.

We chose seven physiologic parameters thought to be the most useful for managing the respiratory and hemodynamic status of patients. These are tabulated, with the units of measurement, in Table . Parameter values were each discretized into five bins; the demarcations were based on meaningful physiologic categories. Thus, for example, the range of Mean arterial pressure (MAP) values was 50,60,70,80,80.1 (mm Hg.). If the $MAP \leq 50$, then the bin was labeled 50, if $MAP> 80$, then the assigned bin was 80.1, and if $50< MAP \leq 60$, the bin assignment was 60, and so forth. Of course, the bin values and/or units are different for each parameter.

| Parameter              | Units       |
|:-----------------------|:------------|
| Mean arterial pressure | mm Hg.      |
| Heart rate             | beats/min   |
| Respiratory rate       | breaths/min |
| pH                     | none        |
| pO2                    | mm Hg.      |
| Pressure volume sensor | cm H2O      |
| Measured flow          | ml/kg-min   |

Study parameters.

# Statistical relational logic models for learning medical policies in ECMO patients

Formally, given EHR data from a set of patients and the set of actions listed in Table , we seek to learn (parameterized) policies for specifying the appropriate medical actions to alter physiologic parameters. In other words, our aim is to learn from the data when physicians should initiate therapy to alter the parameters listed in Table .

| Action                                   |
|:-----------------------------------------|
| Increase mean arterial pressure          |
| Increase/decrease respiratory rate       |
| Decrease heart rate                      |
| Increase/decrease pH                     |
| Increase/decease pO2                     |
| Increase/decrease pressure volume sensor |
| Increase/decrease measured flow          |

Policy actions.

We are inspired by prior work on learning policies using SRL models  where (parameterized weighted logical) clauses were learned from observed trajectories. Broadly known as "imitation learning", the key idea is to learn a distribution over actions such that the policies are as close to the observed user policy as possible. This particular setting is quite useful in cases where the reward function is difficult to specify in advance. Imitation learning algorithms directly optimize the learned policy from trajectories instead of the expected cumulative discounted reward (as in reinforcement learning); in many cases this is easier, since when we have observed trajectories, we can avoid exploration. We consider learning from observations and learn a relational policy from data.

Our SRL learning method is based on learning a set of logical regression (TILDE) trees  in a stage-wise manner. This learning method uses an underlying Inductive Logic Programming (ILP) learner  to induce a set of logical clauses and then fits the weights (parameters) of these clauses. We employ the machinery of gradient-boosting  where differences between observed and predicted probabilities are computed as gradients for the training examples and TILDE trees are learned at each step to fit these gradients. For more details, we refer to our previous work .

Recall that an ILP algorithm accepts a set of facts, sets of positive and negative examples of the concepts to learn, returning logic programs defining the learned concepts. To learn the concepts listed in Table , we include as facts the values of each parameter for each subject for each hour; for instance, "map(subj1,100,70)" represents that subject1 at time step 100 hours had a mean arterial blood pressure between 60-70 mmHg. As noted earlier, not every parameter was measured each hour. The examples were derived from these facts. If on the consideration of two consecutive measurements, there was a significant change in the parameter value (defined as a change of at least two bins in the discretized values), then we generated a positive example. For example, if in addition to the fact listed above, there was "map(subj1,101,80.1)", indicating a significant increase in blood pressure after hour 100, the positive example "mapincr(subj1,100)" would be generated. Otherwise, we synthesized a (false) negative example.

# Results

In this preliminary experiment, we set the parameters of the boosted learning algorithm so that each concept was approximated by a set of 20 relational regression trees. In these trees, each node consists of a logic clause whose possible truth values are represented by the edges. Leaves of the tree are labelled with the weight (and the value subjected to the $sigmoid$ function in parentheses, where $sigmoid(x)=1/1+e^{-x}$) corresponding to logic rules constructed by following from the root to the leaf. A representative probabilistic logic tree is presented in Figure .

We extracted weighted first-order logic rules from this tree; the generated clauses are listed in Table .

```latex
\begin{table*}
%\tiny
\centering
\caption{Weighted first-order logic rules for mean arterial pressure increase generated from a representative boosted tree. Variable A represents the subject. Variables B and C represent the time.}\label{tab:mapActionClauses}
%\setlength{\extrarowheight}{2pt}
\begin{tabular}{|c|l|l|}
% \begin{comment}
\hline
 No. & Wt. &  \multicolumn{1}{c}{Logic rule}\vline\\
\hline
1 & 0.112 & \( \mathtt{mapincr(A,B)}\,\Leftarrow \mathtt{\neg map(A,B,60-70)}\)\\
2 & 0.532 & \( \mathtt{mapincr(A,B)}\,\Leftarrow \mathtt{map(A,B,60-70)} \wedge \mathtt{measured\_flow(A,B,20-50)} \)\\
& & \( \wedge \mathtt{pressure\_volume\_sensor(A,B,>10)}\)\\
% \begin{comment}
3 & 0.095 & \( \mathtt{mapincr(A,B)}\,\Leftarrow\ \mathtt{map(A,B,60-70)}\wedge \neg [\mathtt{measured\_flow(A,B,20-50)} \) \\
& & \( \wedge \mathtt{pressure\_volume\_sensor(A,B,>10)}]\wedge \mathtt{measured\_flow(A,B,100-150)}\)\\
4 & 0.651 & \(\mathtt{mapincr(A,B)}\,\Leftarrow\ \mathtt{map(A,B,60-70)} \wedge\neg[\mathtt{measured\_flow(A,B,20-50)} \) \\
& & \( \wedge \mathtt{pressure\_volume\_sensor(A,B,>10)}]\wedge\neg\mathtt{measured\_flow(A,B,100-150)} \) \\
& & \( \wedge \mathtt{measured\_flow(A,B,50-100)}\wedge \mathtt{resp\_rate(A,B,\leq15)} \) \\
& & \( \wedge \mathtt{resp\_rateincr(A,B)}\)\\
5 & 0.821 & \(\mathtt{mapincr(A,B)}\,\Leftarrow\ \mathtt{map(A,B,60-70)} \wedge\neg[\mathtt{measured\_flow(A,B,20-50)} \) \\
& &\( \wedge \mathtt{pressure\_volume\_sensor(A,B,>10)}]\wedge\neg\mathtt{measured\_flow(A,B,100-150)} \) \\
& &\( \wedge \mathtt{measured\_flow(A,B,50-100)}\wedge \mathtt{resp\_rate(A,B,\leq15)} \) \\
& & \( \wedge \neg\mathtt{resp\_rateincr(A,B)}\wedge\)
\(\mathtt{heart\_rate(A,B,>130)} \) \\
& & \( \wedge [\mathtt{\exists C \mid B = C+1 \wedge resp\_ratedecr(A,C)}]\)\\
6 & 0.069 & \(\mathtt{mapincr(A,B)}\,\Leftarrow\ \mathtt{map(A,B,60-70)} \wedge\neg[\mathtt{measured\_flow(A,B,20-50)} \) \\
& &\( \wedge \mathtt{pressure\_volume\_sensor(A,B,>10)}]\wedge\neg\mathtt{measured\_flow(A,B,100-150)} \) \\
& &\( \wedge \mathtt{measured\_flow(A,B,50-100)}\wedge \mathtt{resp\_rate(A,B,\leq15)} \) \\
& & \(\wedge \neg\mathtt{resp\_rateincr(A,B)}\wedge\)
\(\neg[\mathtt{heart\_rate(A,B,>130)} \) \\
& &\( \wedge [\mathtt{\exists C \mid B = C+1 \wedge resp\_ratedecr(A,C)}]]\)\\
7 & -0.074 & \(\mathtt{mapincr(A,B)}\,\Leftarrow\ \mathtt{map(A,B,60-70)} \wedge\neg[\mathtt{measured\_flow(A,B,20-50)} \) \\
& &\( \wedge \mathtt{pressure\_volume\_sensor(A,B,>10)}]\wedge\neg\mathtt{measured\_flow(A,B,100-150)} \) \\
& &\( \wedge \neg[\mathtt{measured\_flow(A,B,50-100)}\wedge \mathtt{resp\_rate(A,B,\leq15)}]\)\\
& &\( \wedge \mathtt{resp\_rate(A,B,>40)}\wedge\)
\(\mathtt{heart\_rate(A,B,>130)}\)\\
8 & 0.072 & \(\mathtt{mapincr(A,B)}\,\Leftarrow\ \mathtt{map(A,B,60-70)} \wedge\neg[\mathtt{measured\_flow(A,B,20-50)} \) \\
& & \( \wedge \mathtt{pressure\_volume\_sensor(A,B,>10)}]\wedge\neg\mathtt{measured\_flow(A,B,100-150)} \) \\
& &\( \wedge \neg[\mathtt{measured\_flow(A,B,50-100)}\wedge \mathtt{resp\_rate(A,B,\leq15)}]\)\\
& &\( \wedge \neg[\mathtt{resp\_rate(A,B,>40)}\wedge\)
\(\mathtt{heart\_rate(A,B,>130)}]\)\\
& & \( \wedge \mathtt{resp\_rate(A,B,20-30)}\wedge\)
 \(\mathtt{pressure\_volume\_sensor(A,B,0-10)}\)\\
9 & 0.417 & \(\mathtt{mapincr(A,B)}\,\Leftarrow\ \mathtt{map(A,B,60-70)} \wedge\neg[\mathtt{measured\_flow(A,B,20-50)} \) \\
& & \( \wedge \mathtt{pressure\_volume\_sensor(A,B,>10)}]\wedge\neg\mathtt{measured\_flow(A,B,100-150)} \) \\
& &\( \wedge \neg[\mathtt{measured\_flow(A,B,50-100)}\wedge \mathtt{resp\_rate(A,B,\leq15)}]\)\\
& &\( \wedge \neg[\mathtt{resp\_rate(A,B,>40)}\wedge\)
\(\mathtt{heart\_rate(A,B,>130)}]\)\\
& & \( \wedge \neg[\mathtt{resp\_rate(A,B,20-30)}\wedge\)
 \(\mathtt{pressure\_volume\_sensor(A,B,0-10)}]\)\\
\hline
% \end{comment}
\end{tabular}
\end{table*}
```

Owing to the fact that a probabilistic target concept is represented by a sum of the 20 weighted trees, it is difficult to directly interpret the rules generated by our model. However, by comparing some of the weighted rules within a tree, we can elicit findings consistent with known clinical practice. When we look at Rule 1, we see that the weight of the action to increase the mean arterial blood pressure (map) is 0.112 when the map is not between 60-70 mm hg. (which is roughly the normal range). However, comparing to Rule 2, we see that the weight increases to 0.532 when the mean arterial pressure (map) is in the normal range, and if the pump flow is relatively low (20-50 ml/kg-min), and when the pump preload pressure is relatively high ($>10$ cm H2O). The increased weight on this clause, compared to Rule 1, suggests that in circumstances where the map is normal, but if the pump flow is low, physicians may elect to initiate treatment to raise the blood pressure. This is a reasonable treatment maneuver.

We see another example when comparing Rules 5 and 6. The difference between these rules is in the conjunction of the last two clauses: $$\begin{gathered}
    \mathtt{heart\_rate(A,B,>130) \, \wedge} \\
    \mathtt{[\exists \, C \mid B = C+1 \wedge resp\_ratedecr(A,C)]}
\end{gathered}$$ This clause is present in Rule 5 but negated in Rule 6. The markedly elevated $heart\_rate$ ($>130$ beats/minute) and a recent decrease in respiratory rate are clinical signs of disease severity. The higher weight on Rule 5 (when these findings are present) indicates that the presence of these findings will result in a higher probability of the physician moving to increase the map, which is clinically very reasonable.

As we add clauses to the rules or negate them, moving down the tree, it is generally true that the changing weights make clinical sense. That is, a physician is able to explain why the rule was created. However, it is also appears that some clauses seem peripheral to the task of deciding whether to increase the arterial pressure, and would not necessarily be used in the clinical setting. Without question, our automatic system is able to generate longer, more complicated probabilistic logic rules and use more (perhaps obscure, and perhaps important) clinical facts than would a human. Whether such rules will be clinically relevant and useful in a functioning clinical decision support system is a question requiring further research.

# Discussion

We used a statistical relational logic framework to elicit policies for medical management of children undergoing ECMO. Our preliminary results provide some hope that the method can be used to provide interpretable strategies to physicians managing complicated patients that might be useful in automatic clinical decision support systems. To the best of our knowledge, this is the first study using EHR data to learn probabilistic rules governing the management of patients in the hospital. It can be observed from the rules presented that we include existential variables (observations recognized previously); simple encodings into a propositional framework will not suffice for such problems. Instead, a relational framework is necessary. Also, note the weights/regression values on the leaves, demonstrating the need for a "soft" framework and supports our choice of SRL as a natural choice for such modeling tasks.

There are at least a few shortcomings to our study from a clinical perspective. First, it must be acknowledged that we did not have direct access to the physician actions, and rather derived them from the measured physiologic parameters. This complicates our analysis, owing to the fact that we are unable to distinguish when altered physiologic findings are related to medical care or to the course of the underlying disease. It is reasonable to surmise that when we have available the actual physician orders, we will have a cleaner, less noisy, set of data, perhaps allowing greater success in eliciting the medical policies. A second shortcoming is that in this study we selected only a small subset of the recorded parameters- ones thought to be most physiologically significant in the medical decision-making process. We might obtain better results if we broaden the set of parameters. Moreover, in our SRL experiment, we discovered policies encoded in multiple regression trees; without question, one could question whether the trees learned in the boosting algorithm are readily interpretable to physicians managing patients. This is a broader issue; if weighted logic models such as MLNs/PSL etc., are interepretable, then so are these boosted rules. However, we acknowledge that weighted logic may not be as interpretable to domain experts, and there is a need to explore models that are more explainable and interpretable.

Finally, our technique may provide new insight into which physician actions contribute to variation in clinical outcome for children undergoing ECMO support. For example, the most common risks of ECMO include bleeding (related to the necessary anticoagulation of the patient) and neurologic injury- either an intracranial hemorrhage or an ischemic event. What is not known is whether neurologic injury risk can be altered by different management schemes. We surmise that when we evaluate policies for patients partitioned by outcome class (that is, with or without neurologic event), aspects of the policies may be elicited that increase complication risk. We leave this to future work. But if such a finding were confirmed in a clinical study, ECMO outcomes could be improved.

[^1]: We follow the conventional reinforcement learning definition of a policy as a mapping from states to actions

[^2]: This work was presented in part at the 2019 Probabilistic Logic Programming Workshop, Las Cruces NM.