abstract: IoT deployments have been growing manifold, encompassing sensors, networks, edge, fog and cloud resources. Despite the intense interest from researchers and practitioners, most do not have access to large-scale IoT testbeds for validation. Simulation environments that allow analytical modeling are a poor substitute for evaluating software platforms or application workloads in realistic computing environments. Here, we propose VIoLET, a virtual environment for defining and launching large-scale IoT deployments within cloud VMs. It offers a declarative model to specify container-based compute resources that match the performance of the native edge, fog and cloud devices using Docker. These can be inter-connected by complex topologies on which private/public networks, and bandwidth and latency rules are enforced. Users can configure synthetic sensors for data generation on these devices as well. We validate VIoLET for deployments with $>400$ devices and $>1500$ device-cores, and show that the virtual IoT environment closely matches the expected compute and network performance at modest costs. This fills an important gap between IoT simulators and real deployments.
author: Shreyas Badiger, Shrey Baheti and Yogesh Simmhan
*Department of Computational and Data Sciences*
*Indian Institute of Science (IISc), Bangalore 560012, India*
*Email: firstname.lastname@example.com, email@example.com, firstname.lastname@example.com*
bibliography: paper.bib
title: VIoLET: A Large-scale Virtual Environment for Internet of Things [^1] [^2]

# Introduction

Internet of Things (IoT) is expanding rapidly as diverse domains deploy sensors, communication, and gateway infrastructure to support applications such as smart cities, personalized health, and autonomous vehicles. IoT is also accelerating the need for, and the use of edge, fog and cloud resources, in a coordinated manner. The need comes from the availability of large volumes of data streams that need to be analyzed closer to the edge to conserve bandwidth (e.g., video surveillance), or of fast data streams that need to be processed with low latency . Edge gateway devices such as Raspberry Pi and Smart Phones have non-trivial resource capabilities, and can run a full Linux stack on 64-bit ARM processors. Fog devices such as NVidia's TX1 and Dell's Edge Gateways have power-efficient Atom processors or GPUs to support the needs of several edge devices . At the same time, edge and even accelerated fog devices may not have the elastic and seemingly infinite on-demand resource capacity that is available in the cloud, and necessary for processing by certain IoT applications.

Besides production deployments of IoT, there is also active research at the intersection of IoT, and edge, fog and cloud computing that is investigating application scheduling, resiliency, big data platforms, and so on . However, a key gap that exists is the ability to validate these research outcomes on real or realistic IoT environments. Research IoT testbeds may have just 10's of devices, and simulation environments make too many idealized assumptions and do not allow actual applications to be deployed. Manually launching and configuring containers is time consuming and error-prone. Even planning of production deployment of IoT, edge and fog resources are based on analytical models or simulations, which may not hold in practice .

What is lacking is a virtualized IoT environment that offers the computing and network ecosystem of a real deployment without the need to purchase, configure and deploy the edge, fog and networking devices. Here, we propose *VIoLET, a Large-scale Virtual Environment for Internet of Things*. VIoLET offers several essential features that make it valuable for researchers and planners. It is a virtualized environment that uses containers to offer comparable compute resources as edge, fog and cloud, and can run real applications. It allows the easy definition of diverse network topologies, and imposes bandwidth and latency limits between containers. VIoLET also allows the definition of virtual sensors that generate data with various distributions within the containers. It runs on top of cloud VMs or commodity clusters, allowing it to scale to hundreds or thousands of devices, provided cumulative compute capacity is available on the host machines. All of these help setup and validate an environment that mimics the behavior of city-scale IoT deployments in a fast, reproducible and cost-effective manner. *VIoLET v1.0* is available for download from <https://github.com/dream-lab/VIoLET>.

The rest of this paper is organized as follows. We motivate various requirements for VIoLET in Section , describe its architecture design that meets these requirements and its implementation in Section , present results on deploying and scaling VIoLET for different IoT topologies in Section , compare it with related literature and tools in Section , and finally present our conclusions and future work in Section .

# Design Requirements

Here, we present high-level requirements for a *Virtual Environment (VE)* like VIoLET, based on the needs of researchers and developers of applications, platforms and runtime environments for IoT, edge, and fog resources.

**Compute environment.** The VE should provide the ability to configure computing resources that capture the performance behavior of *heterogeneous IoT resources*, such as edge devices, gateways, fog and even cloud resources. Key resource capabilities to be controlled include CPU rating, memory and storage capacity, and network. Further, a *compute environment* that can host platforms and run applications should be provided within these resources. Virtual Machines (VM) have traditionally offered such capabilities, but are too heavy-weight for the often light-weight and plentiful IoT devices. *Containers* are much more light-weight and offer similar capabilities. One downside is the inability to change the underlying Operating System (OS) as it is coupled with the Linux kernel of the host machine. However, we expect most IoT devices to run a flavor of Linux.

**Networking.** Communication is central to IoT, and the networking layer is sensitive to various deployment limitations on the field. Wired, wireless and cellular networks are common, each with different *bandwidth and latency characteristics*. There is also a distinction between *local and wide area networks*, and *public and private networks* – the latter can limit the visibility of devices to each other. These affect the platforms and applications in the computing environment, and can decide who can connect to whom and if an indirection service is required. The VE needs to capture such diverse network topologies and behavior.

**Sensing and Data Streams.** Sensors (and actuators) form the third vital component of IoT. These are often connected to the edge computing devices by physical links, *ad hoc* wireless networks, or even on-board the device. These form the source of the distributed, fast data streams that are intrinsic to IoT deployments. The VE should provide the ability to simulate the generation of sensor event streams with various sampling rates and distributions at the compute devices for consumption by hosted applications.

**Application Environment.** IoT devices often ship with standard platforms and software pre-loaded so that potentially hundreds of devices do not have to be reconfigured across the wide area network. The VE should allow platforms and application environments to be pre-configured as part of the deployment, and the setup to be ready-to-use. Users should not be forced to individually configure each compute resources, though they should have the ability to do so if required.

**Scalable.** IoT deployments can be large in the number of devices and sensors – ranging in the 1000's – and with complex network topologies. A VE should be able to scale to such large deployments with minimal resource and human overheads. At the same time, these devices offer real computing environments that require underlying compute capacities to be available on the host machine(s). Hence, we require the VE to *weakly scale*, as long as the underlying infrastructure provides adequate cumulative compute and network capacity for all the devices. The use of elastic cloud resources as the host can enable this.

**Reproducible.** Simulators offer accurate reproducibility but limit the realism, or the ability to run real applications. Physical deployments are hard to get access to and may suffer from transient variability that affects reproducibility. A VE should offer a balance between running within a realistic deployment while being reproducible at a later point in time. This also allows easy sharing of deployment recipes for accurate comparisons.

**Cost effective.** Clouds are able to offer a lower cost per compute unit due to economies of scale at data centers. But IoT devices while being commodity devices are costlier to purchase, deploy and manage. Having VEs offer comparable resource performance as the IoT deployment but for cheaper compute costs is essential. They should also make efficient use of the pay-as-you-go resources. Further, they should be deployable on-demand on elastic resources and release those resources after the experiments and validations are done.

**Ease of Design and Deployment.** Users should be able to configure large IoT deployments with ease, and have them deploy automatically and rapidly. It should be possible to mimic realistic real-world topologies or generate synthetic ones for testing purposes.

# Architecture

We give the high-level overview architecture of VIoLET first, and then discuss individual components and design decisions subsequently. Fig.  shows the high-level architecture of our framework. Users provide their IoT VE as *JSON deployment documents* (Fig. ) that declaratively capture their requirements. A `devices.json` document lists the devices, their types (e.g., Raspberry Pi 3B, NVidia TX1) and their CPU performance. Another, `sensors.json` document list the virtual sensors and their configurations available. Lastly, the actual deployment document, `deployment.json` lists the number of devices of various types, the network topology of the device inter-connects, including bandwidths and latencies, and optionally the virtual sensors and applications available on each device.

VIoLET takes these documents and determines the number of cloud VMs of a specified type that are required to host containers with resources equivalent to the device types. It also decides the mapping from devices to VMs while meeting the compute capacity, and network bandwidth and latency needs of the topology, relative to what is made available by the host VMs.

Then, containers are configured and launched for each device using *Docker*, and the containers are inter-connected through an overlay network. This allows different private and public networks to be created in the VE. Further, Traffic Control (TC) and Network Address Translation (NAT) rules are set in each container to ensure that the requested network topology, bandwidth and latency limits are enforced.

Virtual sensors, if specified, are then started on each device and their streams available on a local network port in the container. Application environments or startup scripts if specified are also configured or launched. After this, the user is provided with a mapping from the logical device names in their deployment document to the physical device IPs of the matching container, and the VMs on which the containers are placed on. Users can access these devices using the Docker `exec` command. Further, the port numbers at which various logical sensors streams are available on each device is also reported back to the user. Together, these give access to the deployed runtime environment to the user.

## Compute Resources

Containers are emerging as a light-weight alternative to VMs for multi-tenancy within a single host. They use Linux kernel's `cgroups` feature to offer benefits of custom software environment (beyond the OS kernel) and resource allocation and isolation, while having trivial overheads compared to hypervisors. They are well-suited for fine-grained resource partitioning and software sand-boxing among trusted applications.

Computing devices in VIoLET are modeled as containers and managed using the *Docker* automation framework. There are two parts to this: the *resource allocation* and the *software configuration*. Docker allows containers to have resource constraints to be specified [^3]. We use this to limit a container's capacity to match the CPU and Memory available on the native device. We use CPU benchmarks on the native device and the host machine to decide this allocation. The commonly used *CoreMark* [^4] is currently supported for an integer-based workload, while *Whetstone* [^5] has been attempted for floating-point operations. One subtlety is that while we use the multi-core benchmark rating of the device for the CPU scaling, this may map to fewer (faster) cores of the host machine.

A container's software environment is defined by the user as an image script (`Dockerfile`) that specify details like applications, startup services, and environment variables, and allow modular extensibility from other images. Public Docker repositories have existing images for common IoT platforms and applications (e.g., Eclipse Californium CoAP, Microsoft IoT Edge, RabbitMQ, Spark). VIoLET provides a base image that includes its framework configuration and allow users to extend their device images from this base with custom software configuration. This is similar to specifying a VM image, except that the users are limited to the host device's Linux kernel OS [^6]. Hence, defining a compute device in VIoLET requires associating a device type for resources, and a device image for the software environment.

## Network Topology

Users define the network topology for the devices based on three aspects: the public network or a private network the device is part of; the visibility of devices to each other as enforced by firewalls; and the bandwidth and latency between pairs of devices. IoT networks are usually composed of numerous private networks that interface with each other and the public Internet through gateways. We allow users to define logical private networks and assign devices to them. These exist in their own subnet. Each private network has a gateway device defined, and all traffic to the public network from other devices is routed through it. All gateway devices are part of one or more public networks, along with other devices that are on those public networks.

For simplicity, all devices in a private network by default can access each other, and have a common latency and bandwidth specified between pairs of devices by the user; and similarly for all devices connected to a public network. By default, devices on different public networks can reach each other. However, users can override this visibility between any pair of devices, and this is directional, i.e., $D1 \rightarrow D2$ need not imply $D1 \leftarrow D2$.

We implement the bandwidth and latency between devices using *Traffic Control (TC) rules* offered by Linux's `iproute2` utility, and the `network` service that we start on each container using `systemd` [^7]. Here, every unique bandwidth and latency requirement gets mapped to a unique virtual Ethernet port, and the rules and enforced on it. This Ethernet port is also connected to the bridge corresponding to the (private or public) network that the device belongs to. The bridges physically group devices that are on the same network, and also logically assign a shared bandwidth and latency to them. All devices on public networks are also connected to a common `docker-0` bridge for the VM they are present on, and which allows all to all communication by default. Restricting the routing of traffic in a private network to/from the public network only through its gateway device is enacted through `ip` commands and *Network Address Translation (NAT) rules*. These rules redirect packets from the Ethernet port connected to the private network, to the Ethernet port connected to the public network.

Docker makes it easy to define connectivity rules and *IP addressing* of containers present in a single host machine using custom bridges defined on the Docker daemon running on the host. However, devices in VIoLET can be placed on disparate VMs and still be part of the same private network. Such communication between multiple Docker daemons requires custom Docker *overlay networks*. We create a standalone *Docker Swarm pool* which gives us the flexibility to set network and system parameters [^8]. For this, the host machines must be able to access a shared key-value store that maintains the overlay networking information. In VIoLET, we use the *Consul* discovery service as our key-value store that is hosted in a separate container on an *admin VM*.

E.g., Fig.  shows a sample network topology, and the Ethernet ports and bridges to enact this in VIoLET. Here, the edge devices E1.1, E1.2 form a private network PVT-1 with the fog device F1 as a gateway, and likewise E2.1, E2.2 and F2 form another private network, PVT-2. Each device can have sensors enabled to simulate data streams with different distributions. The bandwidth and latency within these private networks is uniform: 100Mbps/0.5ms for PVT-1, and 75Mbps/1ms for PVT-2. F1 and F2 fog devices go on to form a public network PUB-1 along with the cloud device, C1, with 40Mbps/100ms. Similarly, the two cloud devices form another public network PUB-2, with 100Mbps/100ms. All these devices are on a single VM, and the public devices are also connected to the `docker-0` bridge for that VM. While the edge devices are connected to a single overlay network, the fog and cloud devices can be connected to multiple overlay networks, based on bandwidth and latency requirements.

As can be seen, configuring the required network topology is complex and time consuming – if done manually for each IoT deployment. Having a simple declarative document that captures the common network patterns in IoT deployments helps automate this.

## Sensors and Virtual Observation Streams

Edge devices are frequently used to acquire IoT sensor data over hardware interfaces like serial, UART or I2C, and then make them available for applications to process and/or transfer. Experiments and validation of IoT deployments require access to such large-scale sensor data. To enable this, we allow users to define virtual sensors that are collocated with devices. These virtual sensors simulate the generation of sensed events and make them available at a local network port, which acts as a proxy for a hardware interface to the sensor. Applications can connect to this port, read observations and process them as required.

We support various configuration parameters for these sensors. The values for the sensor measurements themselves may be provided either as a text file with real data collected from the field, or as the properties of a statistical distribution, such as uniform random, Gaussian, and Poisson from which we sample and return synthetic values. In addition, the rate at which these values change or the events are generated is also specified by the user. Here too we can specify real relative timestamp or a distribution.

We implement each sensor as a Python script that is launched as part of the container startup. The script starts a *Flask* application server that listens on a local port. It takes the sensor's parameters, and internally starts generating observations corresponding to that. When a client connects to this port and requests a measurement, the service returns the *current* reading. For simplicity, this is reported as a CSV string consisting of a user-defined logical sensor ID, the observation timestamp and a sensed value, but can be easily modified.

## Resource Mapping and Deployment

The *admin VM* runs a service that receives the user's deployment document as a REST request and enacts the deployment on cloud VMs in that data center. The default resource hosts are Amazon EC2 VMs but this can easily be extended to resources on other cloud providers or even a private cluster. All AWS EC2 VM instances belong to a same Virtual Private Cloud (VPC) and the same subnet. On receipt of the deployment request, VIoLET builds a graph of the network topology that is used to deploy the devices onto host resources. Here, the vertices of the graph are the devices and are labeled with the device's CPU requirement, given in the CPU benchmark metrics, e.g., iterations/sec for *CoreMark*, and MWIPS for *Whetstone*. An edge exists if a source device can connect to a sink device, and this is labeled by the bandwidth and latency for that network link. E.g., a private network where all devices can see each other will form a clique.

We then make a gross estimate of the number of underlying resources we require. This is done by adding the vertex weights, dividing by the benchmark metric for the host (cloud VM) and rounding it up. This is the least number of identical host resources, say $n$, needed to meet the compute needs of all devices.

Then, we partition the graph across these $n$ hosts using `gpmetis` such that the vertex weights are balanced across hosts and the sum of edge cuts between hosts, based on device bandwidths, is minimized. This tries to collocate devices with high bandwidth inter-connects on the same host. We then check if the sum of the bandwidth edge cuts between devices in each pair of hosts is less than the available bandwidth capacity between them, and if the sum of benchmark metrics of all devices in a host is smaller than its capacity. If not, we increment $n$ by $1$ and repeat the partitioning, and so on.

This greedy approach provides the least number of host resources and the mapping that will meet the CPU and bandwidth capacities of the deployment. For now, we do not optimize for memory capacity and latency, but these can be extended based on standard multi-parameter optimization techniques.

# Evaluation

We evaluate VIoLET for two different IoT deployment configurations: **D105** with 105 edge and fog devices, and **D408** with 408 edge and fog devices. The configuration of each of the devices, their CoreMark CPU performance and the deployment counts are shown in Table , along with the number of AWS VMs required to support them. CoreMark v1.0 is run with multi-threading enabled.

| *Deployment$\rightarrow$* |  |  | **D105** |  | **D408** |  |
|:---|---:|---:|---:|---:|---:|---:|
| Device | Cores | CMark | Count | $\sum$CMark (k) | Count | $\sum$CMark (k) |
| Pi 2B | 4 | 8,910 | 50 | 445 | 200 | 1,782 |
| Pi 3B | 4 | 13,717 | 50 | 685 | 200 | 2,743 |
| NVidia TX1 | 4 | 26,371 | 4 | 105 | 7 | 184 |
| Softiron | 8 | 76,223 | 1 | 76 | 1 | 76 |
| *Total* |  |  | **1,311** |  | **4,786** |  |
| m4.10XL *(host)* | 40 | 371,384 | 4 | 1,485 | 13 | 4,827 |

Device Perf., Device Counts and Host VM Counts used in Deployments

We use two generations of *Raspberry Pis* as edge devices – *Pi 2B* with $4 \times 900$ MHz ARM32 cores and *Pi 3B* with $4\times1.2$ GHz ARM64 cores, and 1 GB RAM each. In addition, we have two fog resources – a *Softiron 3000 (SI)* with AMD A1100 CPU with $8 \times 2$ GHz ARM64 cores and 16 GB RAM, and an *NVidia TX1* device with $4\times1.7$ GHz ARM64 cores and 4 GB RAM (its GPU is not exposed). We use Amazon AWS *m4.10XL* VMs that have $40 \times 2.4~GHz$ Intel Xeon E5-2676 cores, 160 GB RAM and 10 Gbps network bandwidth as the host. Each costs US\$2.00/hour in the US-East (Ohio) data center. As we see, the D105 deployment with $424$ ARM cores requires $3$ of these VMs with $120$ Xeon cores, and D408 with $1,636$ ARM cores requires $13$ of these VMs with $390$ Xeon cores. These deployments cost about US\$6/hour and US\$26/hour, respectively – these are cheaper than a single Raspberry Pi device, on an hourly basis.

## Results for D105 and D408

The network topology for these two deployments is generated synthetically. D105 is defined with 5 private networks and 4 public networks, while D408 has 8 private networks and 2 public networks. A fog device serves as the gateway in each private network, and we randomly place an equal number of edge devices in each private network. Their respective network configurations are given in Tables  and . Each network has a fixed bandwidth and latency configuration, and this ranges from $5-100$ Mbps bandwidth, and $1-100$ ms latency, as specified. All devices in the public networks can see each other. Edge devices in the private network can access the public network, routed through their gateway, but devices in the public network cannot access the devices in the private network. It takes about 8 mins and 24 mins to launch these two topologies on VIoLET.

| **Network** | **Expected** |  | **Obs. BW Dev.%** |  | **Obs. Lat. Dev.%** |  |
|:---|---:|---:|---:|---:|---:|---:|
| 2-7 | BW (Mbps) | Lat. (ms) | Median | Mean | Median | Mean |
| PVT-1 | 5 | 25 | 11.0 | 11.0 | 0.6 | 0.5 |
| PVT-2 | 5 | 75 | 13.8 | 13.8 | 0.0 | 0.0 |
| PVT-3 | 25 | 1 | 4.8 | 4.8 | 15.0 | 15.5 |
| PVT-4 | 25 | 25 | 4.0 | 3.7 | 1.0 | 1.1 |
| PVT-5 | 25 | 50 | 1.6 | 1.4 | 0.0 | 0.0 |
| PUB-1 | 25 | 75 | -3.6 | -3.6 | 0.0 | 0.0 |
| PUB-2 | 25 | 75 | -3.6 | -3.6 | 0.0 | 0.0 |
| PUB-3 | 25 | 75 | -3.6 | -3.5 | 0.0 | 0.0 |
| PUB-4 | 25 | 75 | -3.6 | -3.6 | 0.0 | 0.0 |

Configuration of private and public networks in D105, and Deviation% between Observed and Expected Bandwidth and Latency per network.

| **Network** | **Expected** |  | **Obs. BW Dev.%** |  | **Obs. Lat. Dev.%** |  |
|:---|---:|---:|---:|---:|---:|---:|
| 2-7 | BW (Mbps) | Lat. (ms) | Median | Mean | Median | Mean |
| PVT-1 | 100 | 5 | -2.6 | -2.4 | 6.0 | 5.2 |
| PVT-2 | 75 | 5 | -1.1 | -1.3 | 3.0 | 4.9 |
| PVT-3 | 75 | 25 | -4.1 | -4.0 | 0.6 | 1.0 |
| PVT-4 | 50 | 5 | 0.0 | 0.1 | 4.0 | 4.9 |
| PVT-5 | 50 | 25 | -1.8 | -2.0 | 0.6 | 0.8 |
| PVT-6 | 25 | 25 | -1.8 | -2.0 | 0.6 | 0.8 |
| PVT-7 | 25 | 5 | 2.8 | 3.2 | 0.6 | 0.8 |
| PVT-8 | 25 | 50 | 4.8 | 5.0 | 0.6 | 0.8 |
| PUB-1 | 25 | 75 | -3.6 | -3.6 | 0.0 | 0.0 |
| PUB-2 | 25 | 100 | -7.0 | -7.0 | 0.0 | 0.0 |

Configuration of private and public networks in D408, and Deviation% between Observed and Expected Bandwidth and Latency per network.

Once deployed, we run four baseline benchmarks to validate them. The first does `fping` between $2n$ pairs of devices in each private and public network, where $n$ is the number of devices in the network, and measures the observed latency on the defined links. Next, we sample a subset of $\frac{n}{2}$ links in each private and public network and run `iperf` on them to measure the observed bandwidth. Since `iperf` is costlier than `fping`, we limit ourselves to fewer samples. Third, we run `traceroute` to verify if the gateway device configured for each device matches the gateway of the private network, as a sanity check. These network sanity checks take $\approx 3~mins$ per network for D105, and run in parallel for all networks. Lastly, we run multi-core CoreMark concurrently on all devices.

Figs.  and   show a violin plot of the *deviation%* of the observed network latency, bandwidth, and CoreMark performance from the expected metrics for the two deployments, where $deviation\% = \frac{(Observed-Expected)}{Expected}\%$. The median value is noted in purple text. We see that the median latency and bandwidth deviation% are within $\pm 5\%$ for both the D105 and D408 deployments, with latency of $0.4\%$ and $1.6\%$, and bandwidth of $4.8\%$ and $-0.8\%$, respectively. This is within the margin of error for even real-world networks. The entire distribution in all these cases does not vary by more than $15\%$, showing a relatively tight grouping given the number of devices and VMs. We analyze these further for diverse network configurations in the next section.

We run the CoreMark CPU benchmark on all the devices concurrently and report the violin plot for the deviation% for each of the $4$ device types. The median CoreMark value for each device is included in the violin, except for the SI fog where we report values from all the trials since there is just one such device in each deployment. We see that for the two Pis and TX1 – the three slowest devices – the median CoreMark deviation% is within $\pm 2.5\%$ for D105, and the most deviation is $+10\%$ for Pi2B. These indicate that the observed performance is marginally higher than expected, and there is little negative deviation for these three devices. However, we see that the single SI fog device, which is the largest device, has a median deviation% of $-42.1\%$ from 40 trials of CoreMark that were run on it. The distribution is also wide, ranging from $-45\%$ to $+21\%$. This indicates that the concurrent multi-threaded CoreMark runs on 10's of containers on the same VM is causing the largest device container to have variable performance. In fact, the sum of the observed CoreMarks for all the deployed devices in D105 is $1,319k$, which is close to the sum of the expected CoreMark from the devices of $1,311k$. So the small over-performance of many small devices is causing the under-performance of the large device. D408 shows a different behavior, with Pi3B showing higher positive deviations, with a median of $23.2\%$, while the other devices show a smaller positive deviation of $2.6$–$6\%$. SI however does show a wider distribution of the deviation% as before.

Besides these baseline network and CPU metrics, we also run two types of application workloads. One of them starts either an MQTT publisher or a subscriber on each device, and each connects to an *Eclipse Mosquitto MQTT broker* on its gateway. A publisher samples observations from a local sensor and publishes it to a unique topic at its gateway broker while a subscriber subscribes to it. This tests the network and process behavior for the common pub-sub pattern seen in IoT. While results are not plotted due to lack of space, we observe that the median end-to-end latency for each message is $\approx 50~ms$, which loosely corresponds to the two network hops required from the publisher to the broker, and broker to subscriber.

Another workload that we evaluate is with the ECHO dataflow platform for edge and cloud . Here, we incrementally launch 100 Extract-Transform-Load dataflows using the Apache NiFi engine on distributed devices and observe the latency time for deployment and the end to end latency for the dataflows. This is yet another use-case for VIoLET to help evaluate the efficacy of such edge, fog and cloud orchestration platforms and schedulers.

## Analysis of Network Behavior

Being able to accurately model network behavior is essential for IoT VEs. Here, we perform more detailed experiments that evaluate the impact of specific bandwidth and latency values on the deviation%. Specifically, we try out 19 different network configurations of the D105 deployment while varying the pair of bandwidth and latency values on these networks. These together form 143 different networks. In Fig. , we plot the deviation% of the mean bandwidth, as the bandwidth increases for different latency values, while in Fig. , we plot the deviation% of the mean latency, as latency increases.

It is clear from Fig.  that the latency deviation is sensitive to the absolute latency value. For small latency values of $1$ ms, the deviation% ranges between $15-40\%$, and this drops to $2.6-8\%$ for $5$ ms. The deviation% exponentially reduces for latencies higher than that, with latencies over $50$ ms having just $0.1\%$ deviation. The latency between VMs is measured at $0.4$ ms, while between containers on the same VM is $0.06$ ms. Hence, achieving a latency better these is not possible, and the achieved latency depends on the placement of containers on the same or different VMs. Since our network partitioning currently is based on bandwidth and compute capacity, and not latency limits, it is possible that two devices requiring low latency are on different VMs. As a result, the deviation% increases. Here, we see that the latency deviation is independent of the bandwidth of the network link.

We observe that the deviation in bandwidth is a function of both latency and bandwidth. In fact, it is also a function of the *TCP window size*, which by default is set to $262,144$ bytes in the containers. The *Bandwidth Delay Product (BDP)* is defined as the product of the bandwidth and latency. For efficient use of the network link, the TCP window size should be greater than this BDP, i.e., $Window \ge Bandwidth \times Latency$. In other words, given a fixed latency and TCP window size, the $Peak~Bandwidth = \frac{Window}{Latency}$.

Fig.  shows the bandwidth deviation% on the Y axis for different latencies, as the bandwidth increases on the X axis. It also shows the maximum possible bandwidth for a given latency (based on the window size) along the bottom X axis. We observe that for low latencies of $1$ – $25$ ms, the bandwidth deviation% is low and falls between $-5.1$ – $18\%$ for all bandwidths from $5$ – $100$ Mbps. This is because with the default window size, even a latency of $25$ ms supports a bandwidth of $83$ Mbps, and lower latencies support an even higher peak bandwidth. The positive deviation% is also high for low bandwidth values and lower for high bandwidth values – even small changes in absolute bandwidth causes a larger change in the relative deviation% when the bandwidth is low.

We also see that as the latency increases, the negative deviation% increases as the bandwidth increases. In particular, as we cross the peak bandwidth value on the X axis, the deviation% becomes more negative. E.g., at $75$ ms, the peak bandwidth supported is only $28$ Mbps, and we see the bandwidth deviation% for this latency worsen from $-3.6\%$ to $-11.9\%$ when the bandwidth configuration increases from $25$ Mbps to $75$ Mbps. This is as expected, and indicates that the users of the container need to tune the TCP window size in the container to enforce bandwidths more accurately.

# Related Work

The growing interest in IoT and edge/fog computing has given rise to several *simulation environments*. *iFogSim*  extends the prior work on CloudSim  to simulate the behavior of applications over fog devices, sensors and actuators that are connected by a network topology. Users define the compute, network and energy profiles of fog devices, and the properties and distributions of tuples from sensors. DAG-based applications with tasks consuming compute capacity and bandwidth can be defined by the user, and its execution over the fog network is simulated using an extensible resource manager. The goal is to evaluate different scheduling strategies synthetically. We similarly let devices, network and sensors to be defined, but actually instantiate the first two – only the sensor stream is simulated. This allows users to evaluate real applications and schedulers.

*Edgecloudsim*  offers similar capabilities, but also introduces mobility models for the edge into the mix. They simulate network characteristics like transmission delay for LAN and WAN, and also task failures due to mobility for a single use-case. *IOTSim*, despite its name, simulates the execution of Map Reduce and stream processing tasks on top of a cloud data center, and uses CloudSim as the base simulation engine. While IoT motivates the synthetic application workloads for their big data platform simulation, they do not actually simulate an IoT deployment.

In the commercial space, city-scale simulators for IoT deployments in smart cities are available . These mimic the behavior of not just devices, sensors, actuators and the network, but also application services like MQTT broker and CoAP services that may be hosted. These offer a comprehensive simulation environment for city-planners to perform what-if analysis on the models. We go a step further and allow realistic devices and networks to be virtualized on elastic cloud VMs, and applications themselves to be executed, without physically deploying the field devices. Simulators are popular in other domains as well, such as cloud, network and SDN simulators .

There have been container-based solutions that are closer to our approach, and allow large-scale customized environments to be launched and applications to be run on them. Ceesay, et al. , deploy container-based environments for Big Data platforms and workloads to test different benchmarks, ease deployment and reduce reporting costs. Others have also used such container-based approaches to inject faults into the containers, and evaluate the behavior of platforms and applications running on them .

Other have proposed IoT data stream and application workloads for evaluating big data platforms, particularly stream processing ones. Here, the sensor data is simulated at large-scales while maintaining realistic distributions . These can be used in place of the synthetic sensor streams that we provide. Our prior work has proposed stream and stream processing application workloads for IoT domains . These can potentially use VIoLET for evaluating execution on edge and fog, besides just cloud resources.

*Google's Kubernetes * is a multi-node orchestration platform for container life-cycle management. It schedules containers across nodes to balance the load, but is not aware of network topologies that are overlaid on the containers. VIoLET uses a simple graph-partitioning approach for placement of containers on VMs to balance the CPU capacity, as measure by CoreMark, and ensure that the required device bandwidths stay within bandwidth available between the hosts.

# Conclusions and Future Work

In this paper, we have proposed the design requirements for a Virtual IoT Environment, and presented VIoLET to meet these needs. VIoLET allows users to declaratively create virtual edge, fog and cloud devices as containers that are connected through user-defined network topologies, and can run real IoT platforms and applications. This offers first-hand knowledge of the performance, scalability and metrics for the user's applications or scheduling algorithms, similar to a real IoT deployment, and at large-scales. It is as simple to deploy and run as a simulation environment, balancing ease and flexibility, with realism and reproducibility on-demand. It is also affordable, costing just US\$26/hour to simulate over $400$ devices on Amazon AWS Cloud. VIoLET serves as an essential tool for IoT researchers to validate their outcomes, and for IoT managers to virtually test various software stacks and network deployment models.

There are several extensions possible to this initial version of VIoLET. One of our limitations is that only devices for which container environments can be launched by Docker are feasible. While any device container that runs a standard Linux kernel using `cgroups` (or even a Windows device [^9])) can be run, this limits the use of edge micro-controllers like Arduino, or wireless IoT motes that run real-time OS. Also, leveraging Docker's support for GPUs in future will help users make use of accelerators present in devices like NVidia TX1 [^10]. There is also the opportunity to pack containers more efficiently to reduce the cloud costs , including over-packing when devices will not be pushed to their full utilization.

Our network configurations focus on the visibility of public and private networks, and the bandwidth and latency of the links. However, it does not yet handle more fine-grained transport characteristics such as collision and packet loss that are present in wireless networks. Introducing variability in bandwidth, latency, link failures, and even CPU dynamism is part of future work. More rigorous evaluation using city-scale models and IoT applications are also planned using large private clusters to evaluate VIoLET's weak scaling.

# Acknowledgments

This work is supported by research grants from VMWare, MHRD and Cargill, and by cloud credits from Amazon AWS and Microsoft Azure. We also thank other DREAM:Lab members, Aakash Khochare and Abhilash Sharma, for design discussions and assistance with experiments. We also thank the reviewers of Euro-Par for their detailed comments that has helped us improve the quality of this paper.

[^1]: To appear in the Proceedings of the 24TH INTERNATIONAL EUROPEAN CONFERENCE ON PARALLEL AND DISTRIBUTED COMPUTING (EURO-PAR), August 27–31, 2018, Turin, Italy, [europar2018.org](https://europar2018.org/program)

[^2]: Selected as a *Distinguished Paper* for presentation at the Plenary Session of the conference

[^3]: Docker Resource Constraints, [docs.docker.com/config/containers/resource_constraints](https://docs.docker.com/config/containers/resource_constraints/)

[^4]: y, Embedded Microprocessor Benchmark Consortium (EEMBC), [coremark.org](http://coremark.org)

[^5]: Whetstone Benchmark History and Results, [roylongbottom.org.uk/whetstone.htm](http://www.roylongbottom.org.uk/whetstone.htm)

[^6]: Docker recently introduced support for Windows and Linux containers hosted on Windows Server using the Hyper-V hypervisor. But this is more heavy-weight than Linux containers, and not used by us currently.

[^7]: Traffic Control in Linux, [tldp.org/HOWTO/Traffic-Control-HOWTO](https://www.tldp.org/HOWTO/Traffic-Control-HOWTO/index.html)

[^8]: Multi-host networking with standalone swarms, [docs.docker.com/network/overlay-standalone.swarm](https://docs.docker.com/network/overlay-standalone.swarm/)

[^9]: Docker for Windows, https://docs.docker.com/docker-for-windows/

[^10]: GPU-enabled Docker Containers, https://github.com/NVIDIA/nvidia-docker


bibliography: lscwp.bib

Astro2020 Science White Paper

Gravitational wave astronomy with LIGO and similar detectors in the next decade

**Thematic Areas:** $\square$ Planetary Systems $\square$ Star and Planet Formation $\XBox$ Formation and Evolution of Compact Objects $\XBox$ Cosmology and Fundamental Physics $\square$ Stars and Stellar Evolution $\square$ Resolved Stellar Populations and their Environments $\square$ Galaxy Evolution $\XBox$ Multi-Messenger Astronomy and Astrophysics

**Principal Author:**

Name: David Shoemaker, LIGO Scientific Collaboration spokesperson Institution: Massachusetts Institute of Technology Email: firstname.lastname@example.com Phone: (617) 253 6411

**Co-authors:** (names and institutions) LIGO Scientific Collaboration; complete author list in https://dcc.ligo.org/LIGO-M1800241

**Abstract :**

We describe the plans for gravitational-wave observations and astrophysics that will be carried out by the LIGO Scientific Collaboration (LSC) in the next decade using data from the LIGO Observatories in the US, and sister facilities abroad in Europe, Japan and India. We provide an overview of gravitational wave signal types that we are targeting, and the role of gravitational waves in time-domain multi-messenger astronomy. We briefly summarize what we can infer from the properties of detected signals, including astrophysical event rates and populations, tests of gravitational-wave properties, highly-dynamical and strong-field tests of General Relativity, probing matter under extreme conditions in neutron stars, and making cosmological measurements with gravitational-wave sources.

The past three years have witnessed the birth of observational gravitational-wave astronomy, starting with the first detection of a binary black hole merger on September 14 2015 , followed by discoveries of nine more in the first and second LIGO/Virgo Observing runs , and the spectacular multi-messenger observation of a merger of neutron stars on August 17, 2017 .

These detections were enabled by a nearly three decade long effort to build Advanced LIGO   comprising two laser interferometric gravitational-wave detectors with suspended mirrors, laser beams traveling in vacuum through 4 km perpendicular arms in each detector, to detect sub-nuclear distance scale changes in distance. The LIGO Scientific Collaboration (LSC) works closely with the Virgo and KAGRA collaborations operating gravitational-wave detectors in Europe and Japan to ensure coordinated observations by the global network.

In this white paper, we describe plans for gravitational-wave observing campaigns and expected science goals in the coming decade. As shown in Fig. , the Advanced LIGO detectors took data between September 2015 and January 2016 in their first Observing run (O1), and then again with improved sensitivity in O2, between November 2016 and August 2017. The Virgo detector joined O2 on August 1st 2017, providing greatly improved sky localization of the detected events. The improved localization and rapid alerts led to the detection of an electromagnetic counterpart to the binary neutron star merger . This counterpart, spanning all bands of the electromagnetic spectrum, allowed the first direct association between a binary neutron star merger and a short gamma-ray burst, and the first unambiguous identification of a kilonova.

In the next few years, the Advanced LIGO and Virgo detectors will continue to observe and analyze data together, and are expected to reach the sensitivity to which they were designed . KAGRA is expected to join in the gravitational wave network in 2019. The GEO600 detector will provide coverage for exceptional events during times when no other detectors will be operating, and will otherwise concentrate on testing technologies for future detectors . The greatest scientific return is possible when all operating GW detectors combine their data.

Funding from the US, UK and Australia has been secured for the "A+" detector upgrade , implementing further sensitivity improvements beyond the current Advanced LIGO design. Toward the middle of the next decade, a new observatory in India will host an Advanced LIGO detector to further enhance the network sensitivity. This decade will see an improvement of a factor of several in astrophysical distance reach, as well as a significant increase in observing time, which translates into a factor of a hundred increase in rate of detections of black holes and neutron stars. However, detection of mergers of more massive black hole at the centers of galaxies will need other instruments, like space detectors and pulsar timing arrays; detections of neutron star mergers at cosmological distances will need a new generation of ground based facilities.

The LSC is working to bring its advanced detectors to their design sensitivity, improve their astrophysical reach with technological advances, and undertake observing runs collecting calibrated gravitational-wave data. It will perform searches for gravitational waves, some in near real time, and low-latency public alerts will be issued to the broader astronomical community to enable multi-messenger observations of gravitational-wave events. The LSC extracts the details of the gravitational-wave signals from the data and, using the measured properties of the signal, presents in publications the astrophysical implications. Following a proprietary period, the LIGO data are made public, enabling other scientists to independently search the data . More details of the LSC's Scientific Goals are available in the LSC Program and accompanying White Papers.

We list in this section the gravitational wave targets that the LSC and Virgo will be searching for in the near term, with KAGRA and LIGO-India joining when possible.

```latex
\begin{wrapfigure}{r}{0.5\textwidth}
\vspace{-25pt}
\begin{center}
   \includegraphics[width=3in]{gw_horizons_2g}
   \end{center}
   \vspace{-15pt}
   \caption{Astrophysical reach of the aLIGO and A+ detectors as a function of total mass of merging binary systems.
   Contours show the maximum reach (Horizon) and distance at which 10\% and 50\% of sources are observable.}
   \label{fig:cbc_range}
   \vspace{-15pt}
\end{wrapfigure}
```

**Gravitational waves emitted during the coalescence of compact binaries**. We will search for mergers of compact binaries that produce gravitational waves in the sensitive frequency range of the LIGO detectors, most prominently binary systems with neutron stars and stellar-mass black holes. These searches are run in low latency to provide alerts to electromagnetic observers.

Binary black hole mergers are expected to be the dominant class of gravitational-wave observations, with A+ having sensitivity to systems with masses up to $1000 M_\odot$ and out to redshifts of $z=3$, as shown in figure . The frequency of binary-black-hole observations is expected to increase to as much as several detections per week by the end of the decade, and we will provide a comprehensive catalog of binary black hole populations during the decade.

Binary neutron stars can be observed to $z\sim 0.1$. With only one detection to date, the astrophysical rate of binary neutron star mergers is not yet well constrained. However, several detections per year are likely. We also search for neutron-star–black-hole binaries, with observations expected within the decade .

**Searches for unmodeled transient gravitational wave signals.** We will search for transients with durations from a few milliseconds up to hours or days . Expected sources include core-collapse supernovae, soft gamma repeaters, neutron star glitches, proto-neutron stars and accretion disks, and cosmic string cusps, as well as high mass strong black hole mergers and eccentric binary systems. Blind, all-sky searches will also allow the discovery of previously unknown sources. Searches for short transients will be run in low latency.

**Gravitational waves associated with known astronomical transients.** We will search for transient gravitational wave signals around known electromagnetic transients such as gamma-ray bursts , fast radio bursts, supernovae , and magnetar flares. Results from multi-messenger observations of these sources will provide insight on their structure and origin. By using the known times and sky locations of these electromagnetic transients and, where applicable, the expected gravitational wave signals, we will perform targeted gravitational wave searches with improved sensitivity over blind all-sky searches. Some of these searches will be performed in low-latency mode to allow for alerts to be issued to the broader community.

**Gravitational waves emitted by previously unknown non-axisymmetric neutron stars.** Neutron stars are invaluable laboratories for physics under extreme conditions. We will search for continuous gravitational wave emission from fast-spinning galactic neutron stars , both isolated and in binary systems. These stars can emit gravitational radiation through a variety of mechanisms, including rotation with elastic deformations, magnetic deformations, unstable r-mode oscillations, and free precession, all of which operate differently in accreting and non-accreting stars. With longer analysis times and improved detector sensitivity, it is likely that new sources in the Milky Way will be localized, although there is still a large uncertainty on possible strength of gravitational wave emission.

**Continuous gravitational waves emitted by known pulsars and other promising sources.** We will search in greater depth for continuous gravitational waves from known sources exploiting astrophysical measurements, such as the frequency evolution of known pulsars and/or the locations of other promising sources, like supernovae and known X-ray binaries . A new generation of telescopes will become available during the next decade, which will increase by an order of magnitude the number of known and monitored pulsars. Improvements in detector sensitivity, longer duration observing runs, new electromagnetic observations, and algorithmic improvements will all contribute to a compelling possibility for detection and observation of continuous gravitational waves. The detection of gravitational radiation from these sources will facilitate decisive progress on one of the most fundamental questions of modern science: the composition and state of matter at extreme densities. It will also allow testing deviations from general relativity, for example, wave polarization or orbital evolution different from that predicted by general relativity.

**Searches for astrophysical and cosmological gravitational wave backgrounds.** We will search for stochastic gravitational-wave backgrounds, formed from the superposition of signals that are individually either too weak or too numerous to resolve . The primary target will be the background from unresolved binary mergers of stellar-mass black holes and/or neutron stars throughout the universe. Rate estimates suggest that this background may be detectable by 2024 , but perhaps earlier using advanced analysis methods. The detection of this background will provide information about stellar-mass binary-black-hole populations at much larger distances than those accessible for the resolved mergers. It will also complement searches for gravitational-wave backgrounds at much lower frequencies, complementing likely detections before the end of the decade by pulsar-timing array and possibly by CMB searches.

We will also search for gravitational-wave backgrounds produced by cosmic string networks, and other backgrounds of cosmological origin . We will search for anisotropies in the background, where the anisotropy may be correlated with large-scale structure, and deviations from general relativity, allowing for non-standard polarizations of the component gravitational waves . After the initial detections, it will be important to extract the properties of the sources that give rise to these backgrounds.

The following list describes the measurements to be carried out for gravitational wave detections and potential conclusions to be drawn from non-detections.

**Signal Characterization.** We will extract the physical properties of the observed gravitational wave signals. For events where the source is well modeled, such as a binary merger, we will extract the physical parameters of the source . Where the signal morphology is not well modeled, such as a core-collapse supernova, we will reconstruct the waveforms. Where possible, we will determine best-fit maps of sky position and distance. For binary mergers, the estimated masses, spin angular momenta, orbital eccentricities and distances will shed light on their (poorly understood) astrophysical formation mechanism . The tidal deformation of neutron stars will contain imprints of the dense nuclear equation of state . Binary merger signals will potentially contain interesting effects such as spin precession and higher order modes. Detection of galactic supernova signals will allow us to probe the core-collapse supernova mechanism while detection of the astrophysical stochastic gravitational-wave background will constrain the star formation history.

**Public alerts.** Starting with the third Observing run (O3) in 2019, LIGO/Virgo alerts will be public. We will issue prompt and open public alerts of significant gravitational wave events in O3 to allow for follow-up observations with electromagnetic and neutrino observatories. Alerts will be distributed for compact binary coalescence (CBC) events including binary black hole, binary neutron star, and neutron star black hole systems as well as unmodeled transient gravitational wave signals. We maintain the *LIGO/Virgo Public Alerts User Guide* for both professional astronomers and science enthusiasts who are interested in receiving alerts and real-time data products related to gravitational wave events. Alerts will be distributed through NASA's Gamma-ray Coordinates Network (GCN). The user guide provides a brief overview of the procedures for vetting and sending alerts, describes their contents and format, and includes instructions and sample code for receiving GCN Notices and decoding sky maps.

**Astrophysical rates and populations.** We will use the observed individual events, primarily compact binary coalescences of black holes and neutron stars, to determine the underlying population of sources in the universe, taking into account selection effects. For binary mergers, this will enable inferences of the distribution of binary masses, including any mass gaps, spin magnitudes and orientations . We will interpret the detected populations in terms of existing models of compact binary formation and evolution . When other sources are observed, we will similarly constrain the underlying astrophysical populations, and where no sources are observed, rate upper limits will be used to constrain the astrophysical population properties. We will determine the implications of stochastic background search results for various cosmological and astrophysical models, including models based on cosmic string cusps and kinks, inflationary scenarios and models due to mergers of binary neutron stars and/or black holes.

**Tests of gravitational-wave properties.** In General Relativity (GR), gravitational waves propagate at a constant speed, independent of frequency, equal to the speed of light. Gravitational wave observations, both with and without electromagnetic counterparts, can be used to look for variations of the speed of gravity (either from the speed of light or as a function of gravitational wave frequency), and Lorentz and parity violations . Observations of gravitational wave transients or stochastic gravitational wave backgrounds in a network of detectors , or of continuous gravitational waves in one or more detectors, allow us to probe the polarization content of the signal and look for the existence of more polarizations than predicted by GR.

**Highly-dynamical and strong-field tests of General Relativity.** Gravitational wave observations from binary coalescences and inference of source properties are made through waveform models built solving the Einstein's field equations. Waveform models can also be built in theories of gravity alternative to GR, where gravitational interactions are mediated by extra degrees of freedom — for example scalar fields. Thus, gravitational wave observations allow us to test for deviations from GR , including violations of the strong equivalence principle. These violations may occur during the early inspiral, as deviations from the post-Newtonian coefficients computed in GR, and also during the merger and ringdown, where the signal may differ from GR predictions in numerical relativity simulations, and from quasi normal modes of black holes in GR. The latter may occur if black hole's uniqueness theorems are violated or the compact object has a "surface" instead of an horizon. We will search for these possible deviations, both in individual signals and by coherently analyzing the population of observed signals.

**Probing extremes of matter.** The detection of continuous gravitational waves from neutron stars, and further observations of neutron star mergers, will give us insight into the ultra-dense interiors of neutron stars. Observations of continuous waves will provide clues about neutron star formation or accretion and magnetosphere physics (from observation of binaries), their spin, thermal and magnetic field evolution, the nature of cold dense matter, and phase transitions in QCD. Electromagnetic observations of the star could be especially helpful in establishing distance, and hence absolute signal strength, and in relating potential electromagnetic pulse phase to gravitational wave signal phase, which is relevant to interpreting the neutron star non-axisymmetry.

The many expected observations of neutron stars in binary mergers, especially if we extract details of the inspiral and the post-merger waveforms, will allow us to probe the neutron-star equation of state . Since the coalescence of binary systems involving neutron stars produces electromagnetic waves, combining electromagnetic and gravitational wave observations can also yield insight into the mechanisms for prompt and post-merger electromagnetic emission . In the fortunate event of a nearby supernova \[probability of O(10%) by 2030\], combining neutrino and gravitational wave observations can yield insight into the collapse and explosion mechanisms.

**Gravitational wave cosmology.** The gravitational waveform emitted during a binary merger can be used to obtain a measurement of the luminosity distance to the binary, known as the standard siren method. Thus, given an accurate measurement of the source redshift it is possible to probe the expansion history of the universe and, concretely, to measure the Hubble constant . The redshift measurement can either be from an electromagnetic observation, directly from the properties of the gravitational wave signal (*e.g.*, merger physics in neutron star mergers) or statistically derived from overlaying a galaxy catalog on the source localization. We expect that the many detections in the coming decade will lead to an era of precision gravitational-wave cosmology.

We have described the exciting science that will be carried out in the next decade using data from a ground-based gravitational wave observatory network including the LIGO Observatories and sister facilities abroad in Europe with improving sensitivities, and then with a larger network including facilities in Japan and India. From the properties of many signals to be detected with the network, we will infer astrophysical event rates and populations, perform tests of gravitational-wave properties, highly-dynamical and strong-field tests of General Relativity, probe matter under extreme conditions in neutron stars, make cosmological measurements with gravitational-wave sources - and probably have new astronomical milestone discoveries.


abstract: 'Health utilities' measure patient preferences for perfect health compared to specific unhealthy states, such as asthma, a fractured hip, or colon cancer. When integrated over time, these estimations are called quality adjusted life years (QALYs). Until now, characterizing health utilities (HUs) required detailed patient interviews or written surveys. While reliable and specific, this data remained costly due to efforts to locate, enlist and coordinate participants. Thus the scope, context and temporality of diseases examined has remained limited.
  .
  Now that more than a billion people use social media, we propose a novel strategy: use natural language processing to analyze public online conversations for signals of the *severity* of medical conditions and correlate these to known HUs using machine learning. In this work, we filter a dataset that originally contained 2 billion tweets for relevant content on 60 diseases. Using this data, our algorithm successfully distinguished mild from severe diseases, which had previously been categorized only by traditional techniques. This represents progress towards two related applications: first, predicting HUs where such information is nonexistent; and second, (where rich HU data already exists) estimating temporal or geographic patterns of disease severity through data mining.
author: CL Thompson; Josh Introne; Clint Young
bibliography: bibtex_HU.bib
date: 2024-10-02
title:  Determining Health Utilities through
  Data Mining of Social Media

# Full Text:

The game theorist John Von Neumann and his collaborator, Morgenstern, designed one of the earliest measures of health utility (HU). Their method, called the Standard Gamble, quantifies quality-of-life by first asking patients to make a hypothetical, important decision. Ultimately, the information contained in that decision is converted to a number ranging between 0 and 1. To start, a patient imagines that researchers have developed a potent drug that can cure the patient's disease. However the pill has a terrible side effect in some patients– instant death. So the patient is asked to decide on the maximum acceptable risk (m), expressed as a probability, that would allow proceeding with the therapy. A HU (u) is defined as 1 - m = u.

```latex
\begin{wrapfigure}{r}{0.45\textwidth}
\vspace{-5mm}
\captionsetup{font=scriptsize}
\includegraphics[width = .45\textwidth]{SG.png}
\caption{Standard Gamble}
%\end{center}
\vspace{-4.8mm}
\end{wrapfigure}
```

Patients with a lower quality of life (lower u) have a stronger desire for health and accommodate more risk (higher m). Importantly, health utilities are not disease-specific, so diverse health conditions and outcomes can be compared. For example, we can say that epilepsy is worse than asthma because, on average, epilepsy patients accept a higher risk of death when faced with the dilemma in the Standard Gamble. In contrast, there is no clear hierarchy for *disease-specific* endpoints such as seizure frequency or pulmonary function.

The integral of a HU, with respect to time, is called a Quality Adjusted Life Year (QALY). This is a metric widely used in cost-utility analyses which assess relative economic value of various health interventions. Therefore, thousands of published studies (including clinical guidelines and CDC assessments of public health) rely on estimates of disease severity in the form of HUs.

For decades, questionnaires or interviews like the Standard Gamble have served to elicit health utilities. Other examples in this class include: Time Trade Off and the Health Utilities Index. An alternative involves using algorithms that map clinical data to HUs. Generally, for a given health condition, clinical data and HU data must be collected simultaneously; then a regression model is built such that the objective data predicts the HU. For example, epilepsy patients with more frequent seizures may report lower quality-of-life, (although some evidence confounds this assumption). These mappings are generally limited to their disease context. For example, knowledge about the relationship of seizure frequency to HU does not transfer to a mapping of pulmonary function to HU.

Most recently, investigators have utilized natural language processing to filter Twitter and other social networks for novel indices of disease severity. Generally, HUs have not been utilized as training labels for these metrics, and algorithms apply to specific conditions only, such as toothaches and depression. Despite vast repositories of patient generated data, there is no general data mining approach that can estimate a HU for any given disease.

Substantial progress has been made by Parimbelli *et al.* who have used sentiment analysis (SA) of online health messaging boards to derive HUs. SA uses natural language processing (NLP) to extract emotional information from online messages; corporations have used it to monitor brand appeal and word-of-mouth recommendations by consumers. Parimbelli *et al.* use NLP to identify relevant content within five wellness domains (e.g. mobility, presence of pain, *etc.*). Then, using SA, a negativity score is created for each domain, providing weights for its HU estimate. The application computes a final score by weighting these HUs with traditionally derived HUs. So far the system has been validated on a scenario related to atrial fibrillation.

Like Parimbelli, we have an ambitious scope: we train an algorithm to estimate a HU for potentially any disease– not just a mapping that transforms disease-specific measurements into HUs. In contrast to Parimbelli, we use alternate natural language features to estimate HUs. We also highlight the performance of our method, comparing its estimates to held out data on 20 traditionally-derived HUs.

```latex
\begin{wrapfigure}{l}{0.27\textwidth}
\vspace{-4mm}
\captionsetup{font=scriptsize}
\includegraphics[width = .27\textwidth]{process3.png}
\caption{Process}
%\end{center}
\vspace{-5.7mm}
\end{wrapfigure}
```

To test the idea that HUs can be derived from online health conversations, we attempted to build an algorithm to classify mild versus severe health conditions– the most essential dichotomy. We began by compiling a list of diseases with HUs previously determined by well-designed, highly-cited studies utilizing the traditional approach of interviews or surveys.

We preferred highly cited studies with the most scope in order to reduce the variation that inevitably arises between HU studies. Some studies undertake broad data collection efforts, apply a uniform methodology and report HUs for more than 100 diseases. Also, we required sufficient specificity with respect to the health conditions studied. Some HU studies aggregate conditions by International Classification of Disease code. Unfortunately, this lossy process results in terms that are too broad or vague for our purposes, such as "esophageal problem."

For selected studies, we extracted health conditions with their HUs as 'gold-standard' labels. The conditions with the lowest HU became the 'severe' set, while the upper half became the 'mild' conditions, although we culled a margin of middle range of HUs (see table 2 below). Then for each condition in each class, we found related social media messages. We performed the search for related messages by using the exact term cited in the study, and then expanded our query to include all matching synonyms as provided by a consumer health vocabulary containing validated mappings between expert and layperson terminology. We defined the entire corpora of social media messages related to a single disease, along with its class label, as 'an example'.

Our social media content came from Twitter, a 'microblogging' site that limits user messages (called, 'tweets') to 140 characters. Challenges of using Twitter include its unmoderated content, unbounded by topic or context. Yet, users generate more than 500 million tweets every day and Twitter data has been utilized for hundreds of epidemiological studies. Data has been published showing that people share highly personal information. We obtained a dataset of 11 million health-related tweets, previously filtered from a dataset of two billion tweets.

Finally we followed a standard methodology for machine learning experiments: each illness was randomly assigned to training, validation, and test sets. All sets had roughly the same proportion of mild and severe illnesses.

To develop a candidate set of features, we took two complementary approaches. For the first approach, we inspected a small sample of tweets. Scrolling through the data revealed that topics related to mild health problems often included a frivolous style such as usage of emoticons, while severe diseases engendered a more formal, grave tone that included conventional punctuation and syntax. Therefore, our candidate features became emoticons and patterns related to punctuation or capitalization.

<img src="PMT.png" alt="image" />

For the second approach, we relied on large scale screening to select single words (unigrams, hereafter). As a convenient source, we took a sentiment dictionary containing 2477 unigrams used for sentiment analysis (SA), then found the frequency of these unigrams in our social media corpus. Selecting the most frequent 50 unigrams, we further refined this set to include 16 unigrams that best separated the classes.

While we screened more than 60 variables for relevance, we selected only 24 features for training, which included 16 unigrams, three emoticon features, and five formality patterns. Using backward selection, we eliminated 15 features that had a minimal contribution to the classification accuracy on the validation set, leaving nine features in the final algorithm.

```latex
\begin{wraptable}{r}{0.65\textwidth}
\vspace{-4mm}
\captionsetup{font=scriptsize}
\includegraphics[width = .65\textwidth]{FFT.png}
\caption{Health Conditions}
%\end{center}
\vspace{-6mm}
\end{wraptable}
```

For our learning algorithms, we compared the following classifiers implemented in the Python library, Scikit-learn: logistic regression, a decision tree, a linear support vector machine, and a Random Forest. These algorithms represent widely known, general-purpose algorithms in machine learning.

The decision tree classifier (with default settings) performed the best of the four algorithms: its accuracy is 72% on the test set. The final algorithm includes nine features, including emoticons, unigrams, and patterns indicating formality of sentence construction. The algorithm tends to misclassify mild diseases as severe. Amid these misclassified examples, there are no obvious trends in terms of data characteristics, such as number of tweets in the test set, nor are there obvious clinical features linking these diseases, such as age of onset, duration, or gender distribution.

Thus we have shown modest accuracy despite several challenges: our social media content is wide-ranging, and users are only a subset of the population. We did not extensively filter tweets to narrow context, such as for self-experience versus unrelated content. Also, the volume of tweets on each health condition is quite variable, ranging from about 100 tweets to 200,000. Since we had to manually extract suitable HUs from the literature, we have a limited supply of examples for training and validation.

```latex
\begin{wraptable}{l}{0.35\textwidth}
\vspace{-3mm}
\captionsetup{font=scriptsize}
\includegraphics[width = .35\textwidth]{Algorithm_table.png}
\caption{Accuracy}
%\end{center}
\vspace{-3mm}
\end{wraptable}
```

Limitations of data mining include the following: as noted in similar studies of online patient generated data, periodic crowd-level events cause unpredictability. Using a diverse set of predictive features, and filtering for specific contexts could mitigate this to some degree. Also, attention will have to be paid to who is posting messages to social media and how they advocate or otherwise act as a proxy for patients with a particular disease. Sometimes HUs are obtained within highly controlled situations, such as clinical drug trials; clearly data mining cannot replace face-to-face elicitation in these circumstances.

Data mining addresses the many gaps in knowledge that exist (especially for the developing world) because of the expense of the traditional methodology. Our work represents a movement towards an information-age scaling of a clinical, research, and quality improvement task: gauging patient complaints related to health conditions and outcomes. In the business world, data mining of consumer sentiment is already widespread. Epidemiologists have already developed systems for tracking case counts using search engine data, Twitter, or Wikipedia pageviews. Much of this work has focused on case counts but has lacked an assessment of case severity, and therefore misses an aspect of disease burden.

In summary, we have used 11 million health-related tweets and a learning algorithm to classify health conditions as severe or mild, a necessary step in the final goal of placing conditions on a continuum of severity. The power of data mining is that it can be applied to any disease with measurements occurring at any time interval. Unlike a survey, which has been the mainstay for more than 70 years, the marginal cost of analyzing an additional health condition is virtually zero. In particular, this has implications for the developing world, where text messaging and social media have grown increasingly prevalent and yet the assessment of disease burden remains challenging.

# Methods

**Literature Selection:** We used Pubmed and Google Scholar to identify established studies that defined health utilities for multiple, specific health conditions. Most of our HUs were obtained from a single study, sponsored by the National Center for Health Statistics (NCHS). It met our criteria since it included 130 conditions and (as an indicator of acceptance) had 268 citations as of April 23, 2016. This agency, charged with monitoring the health of the US population for the Department of Health and Human Services, conducted the research to validate a novel HU measurement instrument (called HALex). The authors of the NCHS study conclude that HALex has good face validity and high correlations to other canonical studies. A secondary goal was to generate a catalog of HUs useful to studies that must rely on HU information from secondary data sources due to the prohibitive expense of acquiring primary data. We bolstered our HU dataset with other studies cataloging HUs, including two containing pediatric health conditions in order to diversify the conditions studied. A few conditions were included that were not part of studies assessing multiple health conditions simultaneously. To mitigate errors related to methodological variations between studies, we preferred that these single-study conditions had clear class membership (very mild or very severe).
**Selection of Concepts:** Although the NCHS study reports HUs on many specific illnesses, it still contains HUs associated with unhelpful, broader diagnostic bins: examples include 'other extremity paralysis', 'orthopedic impairment-other', 'absent bone/joint', and 'hand or finger impairment'. We used common experience and the medical expertise of one of the authors, CLT, a medical doctor, to filter non-specific terms. In about 30 cases, we could not use a selected concept since we had an insufficient number of tweets (arbitrarily, fewer than 100).
**Selection of Tweets:** We filtered our dataset of 11 million health-related tweets for content related to our research. First, we selected all the tweets containing our concept of interest, matching tweets with a capitalization-insensitive algorithm. A concept is comprised of multiple descriptions (synonyms). For each HU study, we found a summary table listing every health condition measured and its corresponding HU. We extracted the exact health description mentioned in the summary table, and then retrieved all of its descriptions and pertinent word forms as defined by a consumer health vocabulary mentioned above. If a tweet matched more than one concept, it was selected for each concept.
**Privacy:** The health-related tweets obtained for this study contained only the content of the tweet, but we eliminated all other metadata. Because the tweets were intentionally public, this study was exempt from the IRB process. The tweets in table 1 were also public, and have been further modified by us to obscure attribution to a specific user.
**Algorithm Development:** For our learning algorithms, we compared the classifiers as discussed in the body of this article and implemented in the Python library, Scikit-learn. We randomized the examples to training, validation and test sets using stratified randomization, so that each set contained a similar range of health utilities. As discussed, we sequentially explored the decision tree hyperparameters by simply assigning increasingly extreme values while monitoring the effect on the validation error. As discussed above, we selected candidate predictive features via two approaches: manually developing features that capture differences in formality and in the second approach, differentiating frequencies of unigrams from the affective lexicon. Nuances related to the capitalization of these unigrams change their predictive value.

# References

# End Notes

**Supplementary Information** Table comprised of all health conditions used in the study and for each: number of tweets, HU value from literature, source of HU value, partition (training, validation, or test).

**Acknowledgements** The authors are grateful to Michael J. Paul and Mark Dredze for sharing Tweets, Reid Priedhorsky for sharing feedback, and Rebecca Anthony for her editing, critiques, and encouragement.

**Author Contributions:** C.L.T. conceptualized the study. J.I. provided feedback on the study design. C.L.T. and C.Y. performed the analyses. C.L.T. prepared the manuscript, with extensive editing by J.I. and C.Y.

**Author Information:** The authors are affiliated with Michigan State University, (East Lansing, MI 48824, USA), from the following departments: Chris Thompson, MD, MHI is from pediatrics, Clint Young, PhD is from the department of Physics, Astronomy and the National Superconducting Cyclotron Laboratory. Josh Introne, PhD is from Media and Information. The authors declare no competing financial interests. The corresponding author is Christopher L. Thompson.