abstract: Mosquito habitat ranges are projected to expand due to climate change. This investigation aims to identify future mosquito habitats by analyzing preferred ecological conditions of mosquito larvae. After assembling a data set with atmospheric records and larvae observations, a neural network is trained to predict larvae counts from ecological inputs. Time series forecasting is conducted on these variables and climate projections are passed into the initial deep learning model to generate location-specific larvae abundance predictions. The results support the notion of regional ecosystem-driven changes in mosquito spread, with high-elevation regions in particular experiencing an increase in susceptibility to mosquito infestation.
author:
;
;
title: Predicting Future Mosquito Larval Habitats Using Time Series Climate Forecasting and Deep Learning

# Introduction

Mosquito habitats and breeding ranges are expanding globally . Habitat preferences are based on the interaction of several factors, including temperature, humidity, rainfall, elevation, and availability of hosts. Climate change has been identified as a key driving factor for the shifts in mosquito distribution over the past 70 years and is likely to continue to be the chief determinant of mosquito population spread . According to current trends, climate change will lead to major shifts in meteorological variables and land cover distributions, including an increase in average temperature, rising ocean levels, and increased severity of storms and droughts.

Regional changes in climate have allowed for the expansion of mosquito populations to new environments. Mosquitoes bring detrimental vector-borne illnesses, such as the West-Nile virus and dengue. As warmer temperatures are correlated with accelerated mosquito development and illness spread, the necessity to predict potential outbreaks has become a priority .

Mosquito breeding sites require specific ecological conditions. Using habitat patterns such as rainwater pools, riverbed pools, streams, and marshes in conjunction with meteorological data allows for the assembly of larvae development models to forecast conditions that meet the habitable requirements . Using artificial intelligence, predictions can be scaled to adaptable national or global models to identify nuanced relationships between atmospheric variables and mosquito abundance .

Previous research has not integrated these artificial intelligence techniques with region-specific analyses to forecast the particular ecosystems that will become most conducive to mosquito habitation. This investigation harnesses deep learning to develop a mosquito larvae abundance model and to conduct time series climate forecasting, with the goal of pinpointing the most likely locales and ecosystems of future mosquito infestations in the United States.

## Research Overview

The research described in this article was divided into three phases. The first phase involved gathering meteorological data and larvae counts from various locations in the United States and using this data set to create a predictive model for mosquito larvae abundance. The second phase involved extracting time series sequences of the said meteorological variables for specific regions of interest, to allow for the forecasting of environmental conditions. The third phase involved feeding these environmental predictions into the predictive model developed in the first phase to obtain quantitative measurements of mosquito larvae abundance and identify potential breeding spots for mosquitoes.

# Mosquito Larvae Abundance Model

## Data Availability Statement

Mosquito larvae abundance data was obtained from the GLOBE Mosquito Habitat Mapper (MHM) tool in the GLOBE Observer app, a worldwide citizen science app . Meteorological data was obtained from the Weather Underground and Weather WX databases, commercial data providers with historical weather archives available on the web . Historical time series sequences were obtained from the Climate Engine tool . Particularly, the Gridded Surface Meteorological dataset (gridMET), derived from the PRISM satellite and the NLADS project, were used for temperature and precipitation records .

## Data Collection

Mosquito observations are collected using the MHM tool, which records the type of water source in which larvae were observed. Possible categories include still water, flowing water, and water containers. Using a GLOBE data processing tool, mosquito larvae observations made from artificial containers (ovitraps) were filtered and removed to minimize the effect of opportunistic data on the true population densities of mosquito larvae . This data cleaning process was grounded on the assumption that the result would be a valid data, as if gathered in the field for the purpose of analysis, rather than due to opportunistic circumstances. Larvae counts measured on the same dates in the same location were also treated as a singular observation. In each region, the monthly averages of daily maximum, mean, and minimum temperature, as well as precipitation amount and days of precipitation, were collected.[^1] Gathering monthly-averaged data was a reasonable decision due to the lag time that has been shown to exist between environmental phenomena and resulting trends in mosquito abundance . The final data set was represented by 166 locations across the contiguous United States, containing the following numerical features: Average Daily Mean Temperature, Average Daily Maximum Temperature, Average Daily Minimum Temperature, Days of Precipitation, Average Daily Precipitation Amount, Elevation, and Larvae Count.

## Model Architecture

Due to the non-optimized nature of the GLOBE database, statistical transformations of mosquito larvae counts were necessary prior to predictive modeling. Log~10~ transformation was applied to the mosquito larvae counts and other variables were standardized using z-scores.

A deep neural network was assembled for the prediction of mosquito larvae counts. The model contained 6 dense layers with 64 hidden nodes each, followed by an output node. The rectified linear activation unit was used between dense layers. Additionally, dropout layers with a dropout probability of 0.2 were applied between dense layers to regularize the model. The model was trained until convergence using Adam optimization, Xavier weight initialization, and a mini-batch size of 8 examples.

Thirty-five of the oldest data examples were withheld as validation data to gauge whether the model could backcast previous mosquito larvae counts from historical ecological data.

## Model Performance

|         | **Training** | **Validation** |
|:--------|:-------------|:---------------|
| **R**   | 0.888        | 0.489          |
| ***P*** | 1.26E-45     | 1.44E-3        |

Model Metrics: Correlations and Their Statistical Significance

The correlation coefficient between training data labels and model predictions was 0.888. However, the correlation coefficient between validation data and model predictions was 0.488. In other words, the deep neural network was able to understand the intricacies of the training data but fell short when it came to generalizing to unseen data. Though a moderate positive correlation existed between predicted larvae counts and ground-truth larvae counts on validation data, there were instances of large residuals between these values. In practice, positive residuals are more concerning than negative residuals, since a negative residual corresponds to an overestimate of mosquito larvae abundance. This may lead to extra mosquito-prevention precautions being taken when not necessary, which is not directly harmful except from a resource-conservation standpoint. On the other hand, a positive residual implies an underestimate of mosquito larvae abundance, which could lead to a false perception of safety. This may lead to vector-borne disease breakouts due to necessary precautions not being instituted. The predictions of the deep neural network reveal more negative than positive residuals, meaning the model tended to liberally flag locations as containing high mosquito larvae abundances, when in truth they were of less concern.

# Climate Forecasting

## Time Series Analysis

Temperature and precipitation data for the summer months between 1979 and 2021 were acquired using the Climate Engine database for all 48 contiguous states. Measurements were averaged spatiotemporally across the 3-month summertime period from June 22 to September 22 for each state.

Prior to using a Long Short-Term Memory network (LSTM) for time series forecasting, it was discovered that the trends in temperature and precipitation somewhat conformed to a pattern resembling the following periodic function, where $T$ is the target atmospheric variable given the year $t$ since the initial year $t_0$. $$T(t) = \lambda t - e^{-\alpha t}\sin(\theta t)\gamma t^\beta + \phi.$$ The approximate parameters that can be used to estimate the trends are as follows: $$\lambda\approx0.01, \alpha\approx-0.01, \theta\approx0.6, \gamma\approx0.5, \beta\approx0.03, \phi\approx T(t_0).$$ This finding confirmed the validity of a supervised deep learning approach to conduct time-series forecasting.

It was also discovered that minimum and maximum temperature shared a high correlation with mean temperature. In particular, $$\label{eq:2}
T_{min}(t) = T_{mean}(t) - k_{min}, \forall t \in \{t_i\}_{i=0}^n$$ $$\label{eq:3}
T_{max}(t) = T_{mean}(t) + k_{max}, \forall t \in \{t_i\}_{i=0}^n,$$ where $k_{min}$ is a constant of adjustment between the minimum and mean temperature and $k_{max}$ is a constant of adjustment between the maximum and mean temperature. Hence, the climate forecasting task for minimum and maximum temperatures was simplified into the following problem: Find $k$ such that $MAE = \frac{1}{n}\sum_{i=0}^{n}|T(t_i) - S_i|$ is minimized, where $T(t_i)$ is the temperature predicted using the corresponding function above and $S_i$ is the true temperature.

## LSTM Networks

An LSTM network with 32 LSTM units followed by a dense layer was trained on the aforementioned temperature and precipitation sequences to conduct climate forecasting. The model's input length was 20 time steps (corresponding to the past 20 years) and the output length was 10 time steps (corresponding to the next 10 years). Dropout regularization with a dropout probability of 0.2 was applied before the LSTM layer to prevent overfitting. The model was trained until convergence using the same specifications as described in Section .

The following algorithm details the process used to generate future climate predictions, where $m$ is the number of locations, $l$ is the length of the lookback sequence, $p$ is the length of the prediction sequence, $t$ is the number of future sequences (each of length $p$), $f: x \rightarrow y$ is a trained LSTM model, $X$ is the array $\{[X_{11} \dots X_{1l}], [X_{21} \dots X_{2l}], \dots, [X_{m1} \dots X_{ml}]\}$, and $Y$ is the output array.

Average summer temperature and average summer precipitation levels were forecasted until the year 2050. Then, maximum and minimum monthly temperatures were derived from average monthly temperatures using equations and . The monthly days of precipitation for a region could not be forecasted due to the absence of available data on this variable. As a result, due to a strong positive correlation between monthly days of precipitation and average precipitation amount, the former was predicted from the latter using a linear model. Elevation changes were not forecasted; future inputs were simply current mean elevations of each region. These six variables were then passed into the neural network described in Section to predict future mosquito larvae abundance.

# Future Mosquito Habitat Identification

## United States Mosquito Larvae Abundance Forecasts

After conducting climate forecasting for each state, mosquito larvae abundance projections were obtained and displayed in the choropleth map in Fig. 5. Our results show that by 2050, the Rocky Mountain states will emerge as likely locations of mosquito breeding. Namely, Colorado, Utah, Wyoming, and New Mexico will contain higher larvae counts than neighboring states as well as the US average larvae count. This result is interesting considering the said states have high elevations, which is an attribute assumed to have low correlations with mosquito abundance. However, upon closer inspection, these projections align with previous research findings, such as those of Derek-Scasta (2021), who discovered that changing weather variability could shift some mosquito species into higher elevations, causing mosquito densities at high-altitude regions to exceed densities at mid-altitude regions . In addition, states along the Atlantic coastline are projected to contain varying mosquito larvae abundances, but results from Louisiana and New Jersey contrast with results from Florida and Maine, thus preventing the generalization of a relationship between coastlines and mosquito larvae abundance.

## Region of Interest: Texas

Due to the observed correlation between elevation and larvae abundance, the state of Texas was selected as a region of interest for further analysis at the regional level. Texas was chosen since it contains a steep longitudinal elevation gradient ranging from the Rocky Mountains to the Gulf of Mexico coast. Climate forecasting was conducted for each of Texas's ten CONUS climate divisions . Segmentation by climate division permitted a more accurate regional characterization of Texas's diverse ecology, which allowed any observed shifts in larvae counts to be better attributed to a particular ecosystem.

Results for Texas confirmed the patterns observed at the national level. Namely, the two climate divisions with the highest elevation, Trans Pecos and High Plains, were projected to experience the greatest rate of increase in larvae abundance. To understand how ecological variables were correlated with this observation, temperature and precipitation were also mapped, as shown in Fig. 7.

Between 2030 and 2050, precipitation is projected to increase more so in the eastern portion of Texas compared to the southern and western portions. The opposite trend was observed for mean temperature. In fact, western regions were projected to experience the highest rate of temperature increase. The likely relationship between these two factors and larvae abundance is that warmer temperatures in western Texas will intensify and prolong drought conditions, stalling precipitation over the next few decades, while making high-altitude conditions more favorable for the habitation of mosquitoes.

## Analysis of Error

The mosquito larvae abundance model's inability to generalize optimally may have been caused by error in the original GLOBE Mosquito Habitat Mapper data set. Locations with high larvae counts were outliers by nature, causing the model's threshold between underfitting and overfitting to be very steep, hence the suboptimal degree of generalization to validation data. Out of 35 validation data examples, the examples that were underestimates of mosquito larvae abundance mostly corresponded to locations where larvae abundance data was scarce. Hence, the quality of the model could be improved by simply augmenting the size of the data set, which is feasible in the near future due to its nature as a citizen science project.

# Conclusions

This article aims to predict the abundance of mosquito larvae across the United States in the year 2050. To achieve this purpose, a data set consisting of citizen-collected mosquito larvae counts and several accompanying atmospheric and spatiotemporal variables is compiled. Then, atmospheric variables are analyzed to identify the conditions most suited to the habitation of mosquito larvae using a deep learning framework. Next, these variables are forecasted using an LSTM model to project future climatic conditions. Finally, these atmospheric projections are inputted back into the original deep learning model to obtain the desired mosquito larvae abundance predictions.

The results from this experiment support the idea that mosquito spread is largely location and ecosystem-dependent, which points to the benefits of utilizing localized citizen-science observations and conducting regional examinations. One finding was that states along the Rocky Mountain Range, which contain some of the highest elevations around the country, were predicted to have the highest larvae abundance in 2050. This observation was further supported by the case study of Texas, which predicted the greatest change in larvae counts to occur in the high-altitude western region. These results show that the greatest shifts in mosquito larvae abundances will occur in high-altitude locales. The most likely cause of this trend is that an increase in temperatures will render high-altitude regions warm enough for mosquito habitation for the first time. Precipitation and proximity to large bodies of water, however, did not appear to have a generalized correlation with larvae abundance, yielding varying larvae counts across the US. It is likely that unusually large larvae count observations in regions along the coastline were due to the presence of lurking variables such as high population densities rather than due to the coastal location itself.

These findings point to the need for increased resource allocation to high-elevation areas to contain mosquito spread and resultant vector-borne diseases, since these locations are forecasted to become high-risk targets in the future. In addition, because there is a meager presence of mosquitoes in high-altitude regions today, the awareness and containment protocols in these areas regarding mosquitoes are likely lacking, which may lead to grave future consequences if no action is taken.

# Acknowledgment

The authors would like to acknowledge the support of the 2022 Earth Explorers Team, NASA STEM Enhancement in the Earth Sciences (SEES) Virtual High School Internship program, and Dr. Russanne Low. The NASA Earth Science Education Collaborative (NESEC) leads Earth Explorers through an award to the Institute for Global Environmental Strategies, Arlington, VA (NASA Award NNX6AE28A). The SEES High School Summer Intern Program is led by the Texas Space Grant Consortium at the University of Texas at Austin (NASA Award NNX16AB89A).

# Additional Information

CS, JN, and RB compiled data, performed research methodologies, and prepared this article. The authors have made the code used in this research available on [GitHub (click)](https://github.com/csun365/Mosquito-Habitat-Prediction) . Data will be made available upon request.

[^1]: Historical ecological data was only available from airports. Mosquito larvae records from locations with no airports within a 30-mile vicinity were excluded.


abstract: With modern data acquisition devices that work fast and very precise, scientists often face the task of dealing with huge amounts of data. These need to be rapidly processed and stored onto a hard disk. We present a LabVIEW program which reliably streams analog time series of MHz sampling. Its run time has virtually no limitation. We explicitly show how to use the program to extract time series from two experiments: For a photodiode detection system that tracks the position of an optically trapped particle and for a measurement of ionic current through a glass capillary. The program is easy to use and versatile as the input can be any type of analog signal. Also, the data streaming software is simple, highly reliable, and can be easily customized to include, e.g., real-time power spectral analysis and Allan variance noise quantification.
  .
  # Program summary
  .
  *Program title:* TimeSeriesStreaming.vi
  *Catalogue identifier*: TimeSeriesStreaming_v1_0
  *Program summary URL:* http://www.nbi.dk/\~czerwin/TimeSeriesStreaming.html
  *Licensing provisions:* Standard CPC licence, http://cpc.cs.qub.ac.uk/licence/licence.html
  *Distribution format:* .zip / .vi
  *Programming language:* LabVIEW
  *Computer:* Any machine running LabVIEW 8.6 or higher
  *Operating system:* Windows XP and Windows 7
  *RAM:* 60 – 360 Mbyte
  *Memory usage:* Code 40.5 Kbyte, front panel 50.7 Kbyte, block diagram 444.1 Kbyte; total VI file size 77.7 Kbyte
  *Classification:* Data acquisition program
  *Nature of problem:* For numerous scientific and engineering applications, it is highly desirable to have an efficient, reliable, and flexible program to perform data streaming of time series sampled with high frequencies and possibly for long time intervals. This type of data acquisition often produces very large amounts of data not easily streamed onto a computer hard disk using standard methods.
  *Solution:* This LabVIEW program is developed to directly stream any kind of time series onto a hard disk. Due to optimized timing and usage of computational resources, such as multicores and protocols for memory usage, this program provides extremely reliable data acquisition. In particular, the program is optimized to deal with large amounts of data, e.g., taken with high sampling frequencies and over long time intervals. The program can be easily customized for time series analyses.
  *Restrictions:* Only tested in Windows-operating LabVIEW environments, must use TDMS format, acquisition cards must be LabVIEW compatible, driver DAQmx installed.
  *Running time:* as desirable: microseconds to hours
address: Niels Bohr Institute, Blegdamsvej 17, 2100 Copenhagen, Denmark
author: Fabian Czerwinski; Lene B. Oddershede
title: TimeSeriesStreaming.vi:
  LabVIEW program
  for reliable data streaming of large analog time series

# Introduction

Precision experiments where data is acquired with high temporal resolution pose a challenge with respect to streaming and saving the data correctly onto a computer hard disk for further processing . Within the nanoscience and biophysical communities, LabVIEW is often the program of choice for control of data acquisition and streaming . Here, we present a highly reliable and efficient data streaming program in LabVIEW. The program is built into modular blocks with the goal of making the design comprehensible and easily compatible for further customization. Also, user-friendliness has been highly valued and we show how to use the program to stream time series data from two typical nanoscale experiments: One involving optical trapping assays , the other ionic current measurements through glass capillaries .

# Program Overview

## Requirements

A time series can originate from a wide range of physical signals, such as temperature, voltage, current etc. The time series, most often in the form of parallel voltage signals, enter the program through a number of channels of an acquisition card, building the interface between computers and experimental setups. Acquisition cards are available in a broad range for various tasks and quality requirements. We used National Instruments cards NI PCI-6251, NI PCIe-M6251, and NI PCI-M6040 . They are coupled into TimeSeriesStreaming.vi by DAQmx, a LabVIEW-internal driver. As precondition, the acquisition card must be compatible with LabVIEW. This holds either for those that can be installed by National Instrument's *Measurement and Automation Explorer*, or for those supplied with a LabVIEW-compatible driver.

## Main Program

The main program is designed in a modular fashion to offer independent as well as interconnected control of different sources of analog signals. Further, it contains support for data-streaming protocols. The programming architecture combines horizontal modules (acquisition, queuing, streaming) with vertical programming patterns (sequential structure, parallel while loops, multicore processing) in order to assure negligible error rates and optional customization.

The different modules of the program are highlighted each by their background color in Figures  and . The four modules deal with elements that concern **computer specifications** (yellow), **acquisition** (blue), **queuing** (red), and **streaming** (green). Each of the modules functions independently from the others as it communicates through well-defined programming patterns.

The usage of the program will be explained in Section .

#### Programming Patterns

*Multicore processing* is the ability to distribute computational jobs over more than one core, i.e. one CPU. This feature has become available in recent versions of LabVIEW. In TimeSeriesStreaming.vi multicore processing is implemented by assigning each timed loop to a specific core. On the tested systems, the CPU load of an individual core never exceeded 20%. Optimal multicoring was ensured by core assignments (highlighted yellow in Figure ). It could also compensate for occasional interruptions by the Windows XP operating system.

Data acquisition must not be interrupted by waiting times during the streaming process. LabVIEW is optimized for *data flow control*. In TimeSeriesStreaming.vi this is achieved by transporting data packages between different loops exclusively through built-in queues (highlighted red in Figure ). The streaming loop is not executed when the queue is empty. This strategy has proven very powerful, as it allows both loops to run as quickly as possible without potential disturbance by waiting times.

*Parallelizing* allows for parallel execution of computational jobs. Data acquisition is done in one while loop, data streaming in a parallel loop. Very reliable *streaming* is achieved by the powerful data format TDMS (Technical Data Management Streaming, National Instruments). Using the primitive TDMS VIs allows for high performance streaming virtually with no limitation.

# How to use the program

A hands-on introduction to TimeSeriesStreaming.vi is given in this section. The perspective user is guided through the modules in the program's front panel (Figure ). An experienced user could adjust the programming architecture in the block diagram at will (Figure ).

The only **computer specification** that must be set is the processor assignment (highlighted yellow). The user can choose to perform processor assignment on quad-, duo-, or single-cores; or simply choose automatic in which case the program will usually assign the highest ordered cores. However, if the user is aware, e.g., that the Windows operating system is utilizing certain cores, it might be beneficial to assign the cores manually.

The **acquisition** module (highlighted blue) controls mainly the settings regarding the acquisition card for a particular measurement. The desired scan rate \[Hz\] must be given in units of Hertz. The physical channels must be specified. `Dev1/ai0:2` denotes the signals input from Dev1 through the channels $0$, $1$, and $2$ (e.g., $x$, $y$, and $z$ coordinates of a recorded movement). The internal limits of recordable voltage signal minimum \[V\] and maximum \[V\] are set in order to optimize the resolution of the recorded time series. It is advisable to have these settings as close as possible to the extrema of the input time series, though, without cutting any of the data points.

Acquisition cards are equipped with an on-board buffer of a certain size indicated by OnboardBuffer. event interval sets the number of data points in each interval. If this number does not exceed the limit given by OnboardBuffer, the data are optimally passed onto the memory. Therefore, it is recommendable to try to keep event interval smaller than OnboardBuffer.

On left side of the front panel, a graph displays the output of the acquisition card in Volts for all specified channels. The graph Readout AcqCard shows the last package passed to the memory. The values of minimum \[V\] and maximum \[V\] shall be set as vertical axis limits. At any time, the acquisition can be aborted by hitting STOP AcqCard. This will halt the execution of looped functions of the DAQmx driver. The indicators DAQ timeout and DAQ done light up, if one of these two reasons terminates loop and therewith the program.

The graph Readout Queue (highlighted red) displays the last package of data passed through the **queue** to the streaming module. Utilizing a user-specified voltage interval here allows for an on-screen check, e.g., for whether data could be exposed to drift. The indicator Elements in Queue reveals the queue's filling level.

**Streaming** and data storage are controlled by the settings highlighted in green. With number of events the user specifies the total number of event intervals to be acquired. Hence, the total number of data points will be: $$\textsf{total number of points}=\textsf{number of events} * \textsf{event interval}\textnormal{.}$$ how many? counts the processed number of events. The entries spec1 value, spec2 value, and spec3 value are numerical values. They will be stored in the resulting TDMS file in the header to the recorded data. It can regard values the user wishes to keep with the data (e.g., laser power, particle dimensions, ...).

The exact path for the TDMS file with the recorded data must be given. Therefore, replace `[C:\filename.tdms]` by an appropriate entry. Note that an already existing file will be overwritten by default. The program automatically returns the file name and stores it in the header of the TDMS file. title, description, and author can be defined by the user and are also stored in the file header.

If the total number of data points has been streamed, or if the STOP streaming button has been pressed, the program terminates and the indicator streaming done lights up. All data recorded up to that point is available in the user defined TDMS file. Also, TDMS files could be access already during streaming, e.g., from inside another VI.

Potential error messages are shown in the bottom right box.

# Data Format and Benchmarks

#### Data Format

The data recorded with TimeSeriesStreaming.vi is stored in a TDMS file. TDMS is an open-source file format developed by National Instruments. It is a binary format optimized for data streaming, thus, it handles dynamically increasing files correctly. Also, one can access the file already during acquisition. There are three ways to access TDMS: Directly in LabVIEW using the primitive TDMS VIs, by third-party plug-ins, or by the program *Diadem* (National Instruments). For the latter method, it is essential to use *Diadem* version 11.1 or higher. Detailed information on how to use Diadem or how to export TDMS files into third-party products can be found under http://www.ni.com/tdms. Third-party plug-ins exist, e.g., for *Matlab*, *OpenOffice*, and *Excel*. *Origin* can open TDMS files directly.

#### Benchmarks

We tested the performance and stability of TimeSeriesStreaming.vi on three different computers designated for data acquisition. Their specifications are made available through the program summary URL . Since LabVIEW only provides acceptable performance on machines operating Windows, we were limited by the choice of the operating system. All tested CPU architectures (quad-, duo-, and single-core) proved to yield very good acquisition. Though, the highest level of customizability applied to multicore processors. We tested the acquisition cards NI PCI-M6040, NI PCI-6521, and NI PCIe-6521.

```latex
\begin{table*}[htdp]\begin{center}
\begin{tabular}{rrrrrr}
\multicolumn{1}{c}{scan rate}&\multicolumn{1}{c}{number of}&\multicolumn{1}{c}{buffer per}&\multicolumn{1}{c}{max filling}&\multicolumn{1}{c}{TDMS file size}&\multicolumn{1}{c}{error rate}\\
\multicolumn{1}{c}{(kHz)}&\multicolumn{1}{c}{channels}&\multicolumn{1}{c}{channel}&\multicolumn{1}{c}{of queue  }&\multicolumn{1}{c}{(MB/s)}&\\
\hline
1,250   &1          &4096               &0          &9.6                &$< .000$\\
500     &2          &2048               &0          &7.7                &$< .000$\\
250     &4          &1024               &0          &15.3           &$.004$\\
100     &3          &1366               &0          &2.3                &$.001$\\
100     &1          &4096               &2          &2.3                &$.001$\\
22      &6          &683                &5          &6.0                &$.003$\\
22      &3          &1366               &3          &3.0                &$.001$\\
22      &1          &4096               &3          &1.0                &$.001$\\
10      &1          &4096               &8          &0.5                &$< .001$\\
1       &1          &4096               &3          &0.05           &$< .000$\\
\end{tabular}
\caption{Benchmark of \textsc{TimeSeriesStreaming.vi} as performed on a NI PCI-6251 connected to a Windows XP computer running LabVIEW 8.6. Each individual run took 4096~s. Scan rate, number of channels, and buffer per channel were set. Maximum filling, file size, and error rate were determined.}
\end{center}
\label{default}
\end{table*}
```

Table 1 represents the benchmarks for the NI PCI-6251 card with thermal noise as input. Scan rate, number of channels, and buffer per channel were set. Then the program ran for 4096 s and maximum filling and file size were determined. An error was counted when two consecutive 16-bit digits were exactly identical. The error rate was defined as number of errors divided by the number of data points.

In addition, we simulated basically all voltage acquisition devices with the *Measurement and Automation Explorer* and performed benchmarks with TimeSeriesStreaming.vi using the input from the simulated device.

# Examples of Program Applications

Two brief examples of how to use TimeSeriesStreaming.vi are described here. In addition, the program has already been used to reliably stream large time series from experiments involving optical trapping of micron-sized polystyrene spheres , gold nanorods , and quantum dots . Furthermore, we implemented an improved calibration protocol for optical tweezers that made use of the main programming features introduced here .

## Position Recording in Optical Tweezers

The development of TimeSeriesStreaming.vi was prompted by a need to stream large amounts of data from optical trapping assay to a hard disk. The goal was to analyze the noise by means of accuracy measurements . Therefore, positions of a trapped microsphere were recorded at sampling frequencies of up to 100 kHz in the order of hours. A short time series is plotted in Figure A.

Experimental details are provided in Reference . The positions were sampled using a photodiode detection system yielding an output in Volts, which were reformulated in terms of metric distances by a calibration factor. Figure B shows the positional power spectrum, which, when properly analyzed, gives information about the calibration factor as well as the strength of the optical trap . A different type of time series analysis, Allan variance analysis, is excellent for quantifying noise in optical trapping assays , in particular in the low frequency regime, which is not possible through normal variance or power spectral analysis. Figure C shows the Allan deviation of the same trace quantifying the exact accuracy for various measurement intervals. For this type of analysis it is crucial to have long overall measurement time series and reliable streaming of the data onto the hard disk, a requirement met by TimeSeriesStreaming.vi. The modular fashion of the program enables straight forward implementation of similar types of calibration or noise quantification routines.

## Ionic Current through Glass Capillary

The translocation of molecules through solid-state nanopores has drawn a lot of attention in recent years due to the enormous potential they hold for parallel screening of biomolecular solutions . Also, glass capillaries with a diameter of 60 nm could be used to detect DNA folding . Here, we used TimeSeriesStreaming.vi to stream the the ionic current measured onto the computer hard disk. A sketch of the experiment is shown in the lower right of Figure . The measured current values are plotted, and in the upper left there is a zoom-in to illustrate that data acquisition was done at a very high rate (1.25 MHz). In this experiment, the error rate was zero. Hence, time series can be analyzed for events happening within sub-milliseconds, the timescale relevant for protein translocation through a nanopore , or for events happening on the order of minutes, a typical timescale for drift, as also visible on Figure .

# Summary

We developed a LabVIEW program that reliably streams large amounts of data correctly onto a computer hard disk. The program was checked on several individual platforms and showed to perform with a very small error rate. The program is made in a modular fashion with the aim of making it user-friendly and easily customizable. As an example, we showed how to acquire the positions of an optically trapped sphere and how this time series data can be further analyzed. We also demonstrated how the program streamed time series of ionic current measurements. As the program easily handles a broad range of analog inputs, there is a wide range of applications, particularly in biophysical nano-scale experiments as pointed out in the two examples. The source code is freely available at through the CPC Program Library and under the standard CPC license agreement .

# Acknowledgements

We thank Lorenz Steinbock for data acquisition and help on the ionic current measurements, and Oliver Otto as well as Jesper Tholstrup for comments on the manuscript. Also, we acknowledge financial support through the KU excellence program.


abstract: A shape of statistical momentum distribution of hadrons produced in high energy particle collisions closely resembles one observed for a broad variety of phenomena in nature. An attempt was made to understand a genesis of this distribution beyond the context of each particular phenomenon.
author: A.Kropivnitskaya, A.Rostovtsev
Institute for Theoretical and Experimental Physics
Moscow 117218, Russia
E-mail: firstname.lastname@example.com, email@example.com
title: Universalities in hadron production and the maximum entropy principle

A multi-hadron production in high energy hadronic collisions is a complicated phenomenon generally modeled by a dynamic system of hadronizing quarks and gluons with many degrees of freedom and high level of correlations. The properties of produced hadrons at any given interaction cannot be predicted. But statistical properties, energy and momentum averages, correlation functions, and probability density functions show regular behavior. Thus, statistical methods must be applied to understand properties of particle production presented by the experimentally measured distributions.

A statistical model for computing high energy collisions of protons with multiple production of particles has been discussed first by E.Fermi in 1950 . Later on, R.Hagedorn has proposed a statistical thermodynamic model to describe the momentum spectra of particles produced in $pp-$collisions . This model approximates the experimentally measured exponential momentum spectra of hadrons with a Boltzmann-like statistical distribution. With an advance of high energy collision experiments with high statistic accumulated the measured momentum spectra are found to deviate from the exponential form. Namely, at high values of particle's transverse momentum $(P_T)$ the spectrum shows a power-law behaviour. This observation is interpreted as a proof of the underlying QCD dynamics of hadronizing partonic system produced in the particle collisions. Though using QCD one calculates elementary interactions of quarks and gluons at microscopic level, a complexity of the system generally doesn't allow an extension of these calculations to predict the observed particle spectra. In addition, it is observed that the relative rates of different particle production and parameters of the momentum spectra depend on global particle's variables like the mass and spin . Therefore, it is tempting to reconsider a statistical approach to understand the particle's transverse momentum spectra in the whole range.

Firstly, we note that the invariant particle production cross-section is approximated by a damped power-law function of a particle's transverse momentum: $$\frac{d\sigma}{dydP_T^2} \sim \frac{1}{(a_0 + P_T)^n}\,,
\label{pdf}$$ where $a_0$ and $n$ are parameters of the distribution function. For simplicity, we consider only particles produced at central rapidity plateau. These particles do not belong to the fragmentation regions of the colliding beams.

It is interesting to note, the same form of the damped power-law distribution describes a variety of phenomena like a velocity spectrum in turbulent liquid, ion energy distribution in geomagnetic plasma, a distribution of the earthquakes as function of it's magnitude (empirical Gutenberg-Richter law), distributions of the avalanches and landslides, forest fires and solar flares, rains and winds, distributions related to the human activities like earnings and settlements sizes, distribution of inter-trade intervals observed on stock markets, statistical distribution of a number of sexual partners, etc. These distributions arise because the same stochastic process is at work, and this process can be understood beyond the context of each example. Moreover, one can see that for small values of the distributed variable the damped power-law expression is reduced to the exponential statistical distribution. What is a genesis of the power-law and exponential statistical distributions?

The least biased method to obtain statistical distributions, which are realized in the nature was promoted by E.T.Jaynes as Maximum Entropy Principle . This Principle states that the physical observable has a distribution, consistent with given constraints which maximizes the entropy. As an example, the Boltzmann exponential distribution arises naturally from a maximization of Gibbs-Shannon entropy under a constraint on an average value of the distributed variable. The textbook examples of the exponential distributions are the energy of the molecules of an ideal gas within an isolated volume and a distance between two neighboring points with $N-$points randomly spread over a limited interval with length $L$. Indeed, in both examples the average values of the distributed variable are well defined. Namely, due to a conservation of the number of molecules and the total energy in the gas volume their ratio gives a predefined value of average kinetic energy per molecule of gas. Similarly, an average distance between two points is obviously defined by a ratio of $L/(N-1)$. However, generally speaking, the constraint on the average value in most cases has a little sense. Therefore, one needs another suitable form of a constraint which results in the damped power-law distribution while maximizing the Gibbs-Shannon entropy [^1]. This new constraint has to correspond to a new conservation law. There is no obvious known conservation law which could yield the damped power-law distribution. Therefore, we propose here a toy model. In this model each measurement of a physical value $x_i$ of interest corresponds to a gain of some bits of information equal to $ln(a_0+x_i)$. Assume, that an observer is allowed to gain in a set of measurements in average a limited information only. This assumption corresponds to the following constraint $$\Sigma P_i ln(a_0 + x_i) = const\,,
\label{constraint}$$ where $P_i$ is a probability to observe a value $x_i$. Maximization of entropy under the constraint () results in the damped power-law probability distribution as function of $x_i$. It is interesting to note that the above constraint () is reduced to a geometric mean value of terms $(a_0+x_i)$. This makes different events correlated with a correlation strength increasing for smaller $a_0$ values. A conservation of an average information is lively exemplified in a world of fractals. An observer doesn't gain new information about the fractal object structure by changing a scale of resolution. If so, a meaning of the $a_0$ parameter is to define a minimal available information. It is interesting to note, the experimental data show that for hadron transverse momentum spectra the $a_0$ value is close to the mass of produced hadron . In addition to the direct measurements of particle spectra a particle-particle correlation and a broadening of non-Poisson particle multiplicity distribution are likely intrinsically related to the appearance of power-law tails in particle momentum spectra. Therefore, an analysis of their dynamics for varying interaction energies will allow further progress. This makes high statistics data accumulated in high energy particle interactions to be a unique laboratory to study the whole class of the power-law phenomena.

# Acknowledgements

The authors thank the organizing committtee of HSQCD'04 conference for warm hospitality. This work was partially supported by grants RFBR-03-02-17291 and RFBR-04-02-16445.

[^1]: For simplicity here we stick to the Gibbs-Shannon entropy form only. Postulating other forms of entropy, like one used in non-extensive thermodynamics  reveals additional conceptual problems


abstract: Continued reliance on human operators for managing data centers is a major impediment for them from ever reaching extreme dimensions. Large computer systems in general, and data centers in particular, will ultimately be managed using predictive computational and executable models obtained through data-science tools, and at that point, the intervention of humans will be limited to setting high-level goals and policies rather than performing low-level operations. *Data-driven autonomics*, where management and control are based on holistic predictive models that are built and updated using generated data, opens one possible path towards limiting the role of operators in data centers. In this paper, we present a data-science study of a public Google dataset collected in a 12K-node cluster with the goal of building and evaluating a predictive model for node failures. We use BigQuery, the big data SQL platform from the Google Cloud suite, to process massive amounts of data and generate a rich feature set characterizing machine state over time. We describe how an ensemble classifier can be built out of many Random Forest classifiers each trained on these features, to predict if machines will fail in a future 24-hour window. Our evaluation reveals that if we limit false positive rates to 5%, we can achieve true positive rates between 27% and 88% with precision varying between 50% and 72%. We discuss the practicality of including our predictive model as the central component of a data-driven autonomic manager and operating it on-line with live data streams (rather than off-line on data logs). All of the scripts used for BigQuery and classification analyses are publicly available from the authors' website.
author:
bibliography: refs.bib
title: Towards Data-Driven Autonomics in Data Centers

# Introduction

Modern data centers are the engines of the Internet that run e-commerce sites, cloud-based services accessed from mobile devices and power the social networks utilized each day by hundreds of millions of users. Given the pervasiveness of these services in many aspects of our daily lives, continued availability of data centers is critical. And when continued availability is not possible, service degradations and outages need to be foreseen in a timely manner so as to minimize their impact on users. For the most part, current automated data center management tools are limited to low-level infrastructure provisioning, resource allocation, scheduling or monitoring tasks with no predictive capabilities. This leaves the brunt of the problem in detecting and resolving undesired behaviors to armies of operators who continuously monitor streams of data being displayed on monitors. Even at the highly optimistic rate of 26,000 servers managed per staffer[^1], this situation is not sustainable if data centers are ever to reach exascale dimensions. Applying traditional autonomic computing techniques to large data centers is problematic since their complex system characteristics prohibit building a "cause-effect" system model that is essential for closing the control loop. Furthermore, current autonomic computing technologies are reactive and try to steer the system back to desired states only after undesirable states are actually entered — they lack predictive capabilities to anticipate undesirable states in advance so that proactive actions can be taken to avoid them in the first place.

If data centers are the engines of the Internet, then data is their fuel and exhaust. Data centers generate and store vast amounts of data in the form of logs corresponding to various events and errors in the course of their operation. When these computing infrastructure logs are augmented with numerous other internal and external data channels including power supply, cooling, management actions such as software updates, server additions/removal, configuration parameter changes, network topology modifications, or operator actions to modify electrical wiring or change the physical locations of racks/server/storage devices, data centers become ripe to benefit from data science. The grand challenge is to exploit the toolset of modern data science and develop a new generation of autonomics that is *data-driven*, *predictive* and *proactive* based on holistic models that capture a data centre as an *ecosystem* including not only the computer system as such, but also its physical as well as its socio-political environment.

In this paper we present the results of an initial study towards building predictive models for node failures in data centers. The study is based on a recent Google dataset containing workload and scheduler events emitted by the Borg cluster management system  in a cluster of over 12,000 nodes during a one-month period . We employed BigQuery , a big data tool from the Google Cloud Platform that allows running SQL-like queries on massive data, to perform an exploratory feature analysis. This step generated a large number of features at various levels of aggregation suitable for use in a machine learning classifier. The use of BigQuery has allowed us to complete the analysis for large amounts of data (table sizes up to 12TB containing over 100 billion rows) in reasonable amounts of time.

For the classification study, we employed an ensemble that combines the output of multiple *Random Forests* (RF) classifiers, which themselves are ensembles of Decision Trees. RF were employed due to their proven suitability in situations were the number of features is large  and the classes are "unbalanced"  such that one of the classes consists mainly of "rare events" that occur with very low frequency. Although individual RF were better than other classifiers that were considered in our initial tests, they still exhibited limited performance, which prompted us to pursue an *ensemble approach*. While individual trees in RF are based on subsets of features, we used a combination of *bagging* and data *subsampling* to build the RF ensemble and tailor the methodology to this particular dataset. Our ensemble classifier was tested on several days from the trace data, resulting in very good performance on some days (up to 88% true positive rate, TPR, and 5% false positive rate, FPR), and modest performance on other days (minimum of 27% TPR at the same 5% FPR). Precision levels in all cases remained between 50% and 72%. We should note that these results are comparable to other failure prediction studies in the field.

The contributions of our work are severalfold. First, we advocate that modern data centers can be scaled to extreme dimensions only by eliminating reliance on human operators by adopting a new generation of autonomics that is data-driven and based on holistic predictive models. Towards this goal, we provide a failure prediction analysis for a dataset that has been studied extensively in the literature from other perspectives. The model we develop has very promising predictive power and has the potential to form the basis of a data-driven autonomic manager for data centers. Secondly, we propose an ensemble classification methodology tailored to this particular problem where subsampling is combined with bagging and precision-weighted voting to maximize performance. Thirdly, we provide one of the first examples of BigQuery usage in the literature with quantitative evaluation of running times as a function of data size. All of the scripts used for BigQuery and classification analysis are publicly available from our website  under the GNU General Public License.

The rest of this paper is organized as follows. The next section describes the process of building features from the trace data. Section  describes our classification approach while our prediction results are presented in Section . Related work is discussed in Section . In Section  we discuss the issues surrounding the construction of a data-driven autonomic controller based on our predictive model and argue its practicality. Section  concludes the paper.

# Building the feature set with BigQuery

The workload trace published by Google contains several tables monitoring the status of the machines, jobs and tasks during a period of approximately 29 days for a cluster of 12,453 machines. This includes task events (over 100 million records, 17GB uncompressed), which follow the state evolution for each task, and task usage logs (over 1 billion records, 178GB uncompressed), which report the amount of resources per task at approximately 5 minute intervals. We have used the data to compute the overall load and status of different cluster nodes at 5 minute intervals. This resulted in a time series for each machine and feature that spans the entire trace (periods when the machine was "up"). We then proceeded to obtain several features by aggregating measures in the original data. Due to the size of the dataset, this aggregation analysis was performed using BigQuery on the trace data directly from Google Cloud Storage. We used the *bq* command line tool for the entire analysis, and our scripts are available online through our Web site .

From task events, we obtained several time series for each machine with a time resolution of 5 minutes. A total of 7 features were extracted, which count the number of tasks currently *running*, the number of tasks that have *started* in the last 5 minutes and those that have *finished* with different exit statuses — *evicted*, *failed*, *finished normally*, *killed* or *lost*. From task usage data, we obtained 5 additional features (again at 5-minute intervals) measuring the load at machine level in terms of: *CPU*, *memory*, *disk time*, *cycles per instruction* (CPI) and *memory accesses per instruction* (MAI). This resulted in a total of 12 *basic features* that were extracted. For each feature, at each time step we consider the previous 6 time windows (corresponding to the machine status during the last 30 minutes) obtaining 72 features in total (12 basic features $\times$ 6 time windows).

The procedure for obtaining the basic features was extremely fast on the BigQuery platform. For task counts, we started with constructing a table of running tasks, where each row corresponds to one task and includes its start time, end time, end status and the machine it was running on. Starting from this table, we could obtain the time series for each feature for each machine, requiring between 139 and 939 seconds on BigQuery per feature (one separate table per feature was obtained). The features related to machine load were computed by summing over all tasks running on a machine in each time window, requiring between 3585 and 9096 seconds on BigQuery per feature. The increased execution time is due to the increased table sizes (over 1 billion rows). We then performed a JOIN of all above tables to combine the basic features into a single table with 104,197,215 rows (occupying 7GB). For this analysis, our experience allows us to judge BigQuery as being extremely fast; an equivalent computation would have taken months to perform on a regular PC.

| Aggregation |   Average, SD, CV    |   Correlation   |
|:-----------:|:--------------------:|:---------------:|
|     1h      | *166 (all features)* |     45(6.5)     |
|     12h     | *864 (all features)* |   258.8(89.1)   |
|     24h     |     284.6(86.6)      |   395.6(78.9)   |
|     48h     |     593.6(399.2)     |   987.2(590)    |
|     72h     |     726.6(411.5)     | 1055.47(265.23) |
|     96h     |     739.4(319.4)     |  1489.2(805.9)  |

Running times required by BigQuery for obtaining features aggregated over different time windows, for two aggregation types: computing *averages*, *standard deviation* (SD) and *coefficient of variation* (CV) versus computing *correlations*. For 1h and 12h windows, average, SD and CV were computed for all features in a single query. For all other cases, the mean (and standard deviation) of the required times per feature are shown.

A second level of aggregation meant looking at features over longer time windows rather than just the last 5 minutes. At each time step, 3 different statistics — averages, standard deviations and coefficients of variation — were computed for each basic feature obtained at the previous step. This was motivated by the suspicion that not only feature values but also their deviation from the mean could be important in understanding system behavior. Six different running windows of sizes 1, 12, 24, 48, 72 and 96 hours were used to capture behavior at various time resolutions. This resulted in 216 additional features (3 statistics $\times$ 12 features $\times$ 6 window sizes).

In order to generate these aggregated features, a set of intermediate tables were used. For each time point, these tables consisted of the entire set of data points to be averaged. For instance, for 1-hour averages, the table would contain a set of 6 values for each feature and for each time point, showing the evolution of the system over the past hour. While generating these tables was not time consuming (requiring between 197 and 960 seconds), their sizes were quite impressive: ranging from 143 GB (over 1 billion rows) for 1 hour up to 12.5 TB (over 100 billion rows) in the case of 96-hour window. Processing these tables to obtain the aggregated features of interest required significant resources and would not have been possible without the BigQuery platform. Even then, direct queries using a single group by operation to obtain all 216 features was not possible, requiring only one basic feature to be handled at a time and combining the results into a single table at the end. Table  shows statistics over the time required to obtain one feature for the different window sizes.

Although independent feature values are important, another criterion that could be important for prediction is the relations that exist between different measures. Correlation between features is one such measure, with different correlation values indicating changes in system behavior. Hence we introduced a third level of aggregation of the data by computing correlations between a chosen set of feature pairs, again over various window sizes (1 to 96 hours as before). We chose 7 features to analyze: number of running, started and failed jobs together with CPU, memory, disk time and CPI. By computing correlations between all possible pairings of the 7 features, we obtained a total of 21 correlation values for each window size. This introduces 126 additional features to our dataset. The BigQuery analysis started from the same intermediate tables as before and computed correlations for one feature pair at a time. As can be seen in Table , this step was more time consuming, requiring greater time than the previous aggregation step, yet still remains manageable considering the size of the data. The amount of data processed for these queries ranged from 49.6GB (per feature pair for 1-hour windows) to 4.33TB (per feature pair for 96-hour windows), resulting in a higher processing cost (5 USD per TB processed). Yet again, a similar analysis would not have been possible without the BigQuery platform.

The Google trace also reports machine events. These are scheduler events corresponding to machines being added or removed from the pool of resources. Of particular interest are remove events, which can be due to two causes: machine failures or machine software updates. The goal of this work is to predict remove events due to *machine failures*, so the two causes have to be distinguished. Prompted by our discussions, publishers of the Google trace investigated the best way to perform this distinction and suggested to look at the length of time that machines remain down — the time from the remove event of interest to the next add event for the same machine. If this "down time" is large, then we can assume that the remove event was due to a machine failure, while if it is small, the machine was most likely removed to perform a software update. To ensure that an event considered to be a failure is indeed a real failure, we used a relatively-long "down time" threshold of 2 hours, which is greater than the time required for a typical software update. Based on this threshold, out of a total of 8,957 remove events, 2,298 were considered failures, and were the target of our predictive study. For the rest of the events, for which we cannot be sure of the cause, the data points in the preceding 24-hour window were removed completely from the dataset. An alternative would have been considering them part of the safe class, however this might not be true for some of the points. Thus, removing them completely ensures that all data labeled as safe are in fact safe.

To the above features based mostly on load measures, we added two new features: the *up time* for each machine (time since the last corresponding add event) and number of remove events for the entire cluster within the last hour. This resulted in a total of 416 features for 104,197,215 data points (almost 300GB of processed data). Figure  displays the time series for 4 selected features (and the remove events) at one typical machine.

# Classification approach

The features obtained in the previous section were used for classification with the *Random Forest* (RF) classifier. The data points were separated into two classes: safe (negatives) and fail (positives). To do this, for each data point (corresponding to one machine at a certain time point) we computed *time_to_remove* as the time to the next remove event. Then, all points with *time_to_remove* less than 24 hours were assigned to the class fail while all others were assigned to the class safe. We extracted all the fail data points corresponding to real failures (108,365 data points) together with a subset of the safe class, corresponding to 0.5% of the total by random subsampling (544,985 points after subsampling). We used this procedure to deal with the fact that the safe class is much larger than the fail class and classifiers have difficulty learning patterns from very imbalanced datasets. Subsampling is one way of reducing the extent of this imbalance . Even after this subsampling procedure, negatives are about five times the number of positives. These 653,350 data points (safe plus fail) formed the basis of our predictive study.

Given the large number of features, some might be more useful than others, hence we explored two types of feature selection mechanisms. One was *principal component analysis*, which uses the original features to build a set of principal components — additional features that account for most of the variability in the data. Then one can use only the top principal components for classification, since those should contain the most important information. We trained classifiers with an increasing number of principal components, however the performance obtained was not better than using the original features. A second mechanism was to filter the original features based on their correlation to the time to the next failure event (*time_to_remove* above). Correlations were in the interval $[-0.3,0.45]$, and we used only those features with absolute correlation larger than a threshold. We found that the best performance was obtained with a null threshold, which means again using all features. Hence, our attempts to reduce the feature set did not produce better results that the RF trained directly on the original features. One reason for this may be the fact that the RF itself performs feature selection when training the Decision Trees. It appears that the RF mechanism performs better in this case that correlation-based filtering or principal component analysis.

To evaluate the performance of our approach, we employed cross validation. Given the procedure we used to define the two classes, there are multiple data points corresponding to the same failure (data over 24 hours with 5 minutes resolution). Since some of these data points are very similar, choosing the *train* and *test* data cannot be done by selecting random subsets. While random selection may give extremely good prediction results, it is not realistic since we would be using test data which is too similar to the training data. This is why we opted for a time-based separation of train and test data. We considered basing the training on data over a 10-day window, followed by testing based on data over the next day with no overlap with the training data. Hence, the test day started 24 hours after the last training data point. The first two days were omitted in order to decrease the effect on aggregated features. In this manner, fifteen train/test pairs were obtained and used as benchmarks to evaluate our analysis (see Figure ). This forward-in-time cross validation procedure ensures that classification performance is realistic and not an artifact of the structure of the data. Also, it mimics the way failure prediction would be applied in a live data center, where every day a model could be trained on past data to predict future failures.

Given that many points from the fail class are very similar, which is not the case for the safe class due to initial subsampling, the information in the safe class is still overwhelmingly large. This prompted us to further subsample the negative class in order to obtain the training data. This was performed in such a way that the ratio between safe and fail data points is equal to a parameter *fsafe*. We varied this parameter with the values $\{0.25,0.5,1,2,3,4\}$ while using all of the data points from the positive class so as not to miss any useful information. This applied only for training data: for testing we always used all data from both the negative and positive classes (out of the base dataset of 653,350 points). We also used RF of different sizes, with the number of Decision Trees varying from 2 to 15 with a step of 1 (resulting in 14 different values).

As we will discuss in the following section, the performance of the individual classifiers, while better than random, was judged to be not satisfactory. This is why we opted for an ensemble method, which builds a series of classifiers and then selects and combines them to provide the final classification. Ensembles can enhance the power of low performing individual classifiers , especially if these are diverse : if they give false answers on different data points (independent errors), then combining their knowledge can improve accuracy. To create diverse classifiers, one can vary the model parameters but also train them with different data (known as the *bagging* method ). Bagging matches very well with subsampling to overcome the rare events problem, and in fact it has been shown to be effective for the class-imbalance problem . Hence, we adopt a similar approach to build our individual classifiers. Every time a new classifier is trained, a new training dataset is built by considering all the data points in the positive class and a random subset of the negative class. As described earlier, the size of this subset is defined by the *fsafe* parameter. By varying the value of *fsafe* and number of trees in the RF algorithm, we created diverse classifiers. The following algorithm details the procedure used to build the individual classifiers in the ensemble.

We repeated this procedure 5 times, resulting in 5 classifiers for each combination of the parameters *fsafe* and RF size. This resulted in a total of 420 RF in the ensemble (5 repetitions $\times$ 6 *fsafe* values $\times$ 14 RF sizes).

Once the pool of classifiers is obtained, a combining strategy has to be used. Most existing approaches use the majority vote rule — each classifier votes on the class and the majority class becomes the final decision . Alternatively, a weighted vote can be used, and we opted for *precision-weighted voting*. For most existing methods, weights correspond to the accuracy of each classifier on training data . In our case, performance on training data is close to perfect and accuracy is generally high, which is why we use precision on a subset of the test data. Specifically, we divide the test data into two halves: an *individual test* data set and an *ensemble test* data set. The former is used to evaluate the precision of individual classifiers and obtain a weight for their vote. The latter provides the final evaluation of the ensemble. All data corresponding to the test day was used, with no subsampling. Table shows the number of data points used for each benchmark for training and testing. While the parameter *fsafe* controlled the ratio safe/fail during training, fail instances were much less frequent during testing, varying between 13% and 36% of the number of safe instances.

|  | Train | Individual Test |  | Ensemble Test |  |
|:--:|:--:|:--:|:--:|:--:|:--:|
| Benchmark | fail | fail | safe | fail | safe |
|  |  |  |  |  |  |

Size of training and testing datasets. For training data, the number of safe data points is the number of fail multiplied by the *fsafe* parameter at each run.

To perform precision-weighted voting, we first applied each RF $i$ obtained above to the *individual test data* and computed their precision $p_i$ as the fraction of points labeled fail that were actually failures. In other words, *precision* is the probability that an instance labeled as a failure is actually a real failure, which is why we decided to use this as a weight. Then we applied each RF to the *ensemble test data*. For each data point $j$ in this set, each RF provided a classification $o_i^j$ (either 0 or 1 corresponding to safe or fail, respectively). The classification of the ensemble (the whole set of RF) was then computed as a *continuous score* $$s_j=\sum_i{o_i^j p_i}$$ by summing individual answers weighted by their precision. Finally, these were normalized by the highest score in the data $$\label{eq_score}
s_j'=\frac{s_j}{max_j(s_j)}$$ The resulting score $s_j'$ is proportional to the likelihood that a data point is in the fail class — the higher the score, the more certain we are that we have an actual failure. The following algorithm outlines the procedure of obtaining the final ensemble classification scores.

It assumes that the set of classifiers is available (*classifiers*), together with the two test data sets (*individual_test* and *ensemble_test*) and procedures to compute precision of a classifier on a dataset (*Precision*()) and to apply a classifier to a data point (*Classify*() which returns 0 for safe and 1 for fail).

# Classification results

The ensemble classifier was applied to all 15 benchmark datasets. Training was done on an iMac with 3.06GHz Intel Core 2 Duo processor and 8GB of 1067MHz DDR3 memory running OSX 10.9.3. Training of the entire ensemble took between 7 and 9 hours for each benchmark dataset.

Given that the result of the classification is a continuous score (Equation ), and not a discrete label, evaluation was based on the *Receiver Operating Characteristic* (ROC) and *Precision-Recall* (PR) curves. A class can be obtained for a data point $j$ from the score $s_j'$ by using a threshold $s^*$. A data point is considered to be in the fail class if $s_j'\geq s^*$. The smaller $s^*$, the more instances are classified as failures. Thus, by decreasing $s^*$ the number of true positives grows but so do the false positives. Similarly, at different threshold values, a certain precision is obtained. The ROC curve plots the True Positive Rate (TPR) versus the False Positive Rate (FPR) of the classifier as the threshold is varied. Similarly, The PR curve displays the *precision* versus *recall* (equal to TPR or Sensitivity). It is common to evaluate a classifier by computing the *area under ROC* (AUROC) and *area under PR* (AUPR) curves, which can range from 0 to 1. AUROC values greater than 0.5 correspond to classifiers that perform better than random guesses, while AUPR represents an average classification precision, so, again, the higher the better. AUROC and AUPR do not depend on the relative distribution of the two classes, so they are particularly suitable for class-imbalance problems such as the one at hand.

Figure shows AUROC and AUPR values obtained for all datasets, evaluated on the *ensemble test* data. For all benchmarks, AUROC values are very good, over 0.75 and up to 0.97. AUPR ranges between 0.38 and 0.87. Performance appears to increase, especially in terms of precision, towards the end of the trace. Lower performance that is observed for the first two benchmarks could be due to the fact that some of the aggregated features (those over 3 or 4 days) are computed with incomplete data at the beginning.

To evaluate the effect of the different parameters and the ensemble approach, Figure  displays the ROC and PR curves for the benchmarks that result in the worst and best results (4 and 14, respectively). Performance of the individual classifiers in the ensemble are also displayed (as points in the ROC/PR space since their answer is categorical). We can see that individual classifiers result in very low FPR which is very important in predicting failures. Yet, in many cases, the TPR values are also very low. This means that most test data is classified as safe and very few failures are actually identified.

TPR appears to increase when the *fsafe* parameter decreases, but at the expense of the FPR and Precision. The plots show quantitatively the clear dependence between the three plotted measures and *fsafe* values. As the amount of safe training data decreases, the classifiers become less stringent and can identify more failures, which is an important result for this class-imbalance problem. Also, the plot shows clearly that individual classifiers obtained with different values for *fsafe* are diverse, which is critical for obtaining good ensemble performance.

In general, the points corresponding to the *individual classifiers* are below the ROC and PR curves describing the performance of the *ensemble*. This proves that the ensemble method is better than the individual classifiers for this problem, which can be also due to their diversity. Some exceptions do appear (points above the solid lines), however for very low TPR (under 0.2) so in an area of the ROC/PR space that is not interesting from our point of view. We are interested in maximizing the TPR while keeping the FPR at bay. Specifically, the FPR should never grow beyond 5%, which means few false alarms. At this threshold, the two examples from Figure  display TPR values of 0.272 (worst case) and 0.886 (best case), corresponding to precision values of 0.502 and 0.728 respectively. This is much better than individual classifiers at this level, both in terms of precision and TPR. For failure prediction, this means that between 27.2% and 88.6% of failures are identified as such, while from all instances labeled as failures, between 50.2% and 72.8% are actual failures.

In order to analyze the implications of the obtained results in more detail, the relation between the classifier label and the exact time until the next remove event was studied for the data points. This is important because we originally assigned the label safe to all data points that are more than 24 hours away from a failure. According to this classification, a machine would be considered to be in safe state whether it fails in 2 weeks or in 2 days. Similarly, it is considered to be in fail state whether it fails in 23 hours or in 10 minutes. Obviously these are very different situations, and the impact of misclassification varies depending on the time to the next failure. Figure displays this graphically. As the time to the next failure decreases, a safe data point misclassified as fail counts less as a misclassification, since failure is actually approaching. Similarly, a fail data point labeled as safe has a higher negative impact when it is close to the point of failure.

We would like to verify whether our classifier assigns correct and incorrect labels uniformly within each class, irrespective of the real time to the next failure. For this, Figure  shows the distribution of the time-to-the-next-failure, in the form of boxplots, for true positives (TP), false negatives (FN), true negatives (TN) and false positives (FP), again for the worst and best cases (benchmarks 4 and 14, respectively). We look at results obtained at 5% FPR values. A good result would be if misclassified positives are further in time from the point of failure compared to correctly classified failures. And misclassified negatives are closer to the failure point compared to correctly classified negatives.

All positive instances, which are data points corresponding to real failures, are shown in the left panels (Figure and ). These are divided into TP (failures correctly identified by the classifier) and FN (failures missed by the classifier). All data points have a time to the next remove event between 0 and 24 hours, due to the way the positive class was defined. If the classifier was independent of the time to the next failure, the two distributions shown for TP and FN would be very similar. However, the plots show that TP have, on average, lower times until the next event, compared to FN. This means that many of the positive data points that are misclassified are further in time from the actual failure moment compared to those correctly identified. This is good news, because it suggests that although data points are not recognized as imminent failure situations when there is still some time left before the actual failure, correct classification may occur as the failure moment approaches. In fact, if we compute the fraction of failure events that are flagged at least once in the preceding 24 hour window, we obtain values larger than the TPR computed at the data-point level, especially when prediction power is lowest (52.5% vs. 27.2% for benchmark 4 and 88.7% vs. 88.6% for benchmark 14).

Negative (safe) instances — data points that do not precede a remove event by less than 24 hours — can be divided into two classes: those for which there is a remove due to a real failure before the end of the trace and those for which there is none. For the latter, we have no means to estimate the time to the next failure event. So Figures  and show the time to the next remove only for the former, i.e., those machines which will fail before the end of the trace. This is only a small fraction of the entire safe test data, especially for the benchmark with best performance, because it is only three days before the trace ends. However it still provides some indication on the time to the next failure of the negative class, divided into FP (negatives that are labeled as failures) and TN (negatives correctly labeled safe). As the figure shows, on average, times to the next failure are lower for FP compared to TN. This is again a good result because it means that many times the classifier gives false alarms when a failure is approaching, even if it is not strictly in the next 24 hours.

# Related work

The publication of the Google trace data has triggered a flurry of activity within the community including several with goals that are related to ours. Some of these provide general characterization and statistics about the workload and node state for the cluster and identify high levels of heterogeneity and dynamism in the system, especially when compared to grid workloads . User profiles and task usage shapes have also been characterized for this cluster. Other studies have applied clustering techniques for workload characterization, either in terms of jobs and resources or placement constraints , with the aim to synthesize new traces. A different type of usage is for validation of various workload management algorithms. Examples are where the trace is used to evaluate consolidation strategies, where over-committing (overbooking) is validated, who take heterogeneity into account to perform provisioning or investigating checkpointing algorithms.

System modeling and prediction studies using the Google trace data are far fewer than those performing characterization or validation. An early attempt at system modeling based on this trace  validates an event-based simulator using workload parameters extracted from the data, with good performance in simulating job status and overall system load. Host load prediction using a Bayesian classifier was analyzed in . Using CPU and RAM history, the mean load in a future time window is predicted, by dividing possible load levels into 50 discrete states. Here we perform prediction of machine failures for this cluster, which has not been attempted to date, to our knowledge.

Failure prediction has been an active topic for many years, with a comprehensive review presented in . This work summarizes several methods of failure prediction in single machines, clusters, application servers, file systems, hard drives, email servers and clients, etc., dividing them into failure tracking, symptom monitoring or error reporting. The method introduced here falls into the symptom monitoring category, however elements of failure tracking and error reporting are also present through features like number of recent failures and job failure events.

More recent studies concentrate on larger scale distributed systems such as HPC or clouds. For failure tracking methods, an important resource is the failure trace archive , a repository for failure logs and an associated toolkit that can enable integrated characterization of failures, such as distributions of inter-event times. Job failure in a cloud setting has been analyzed in . The naive Bayes classifier is used to obtain a probability of failure based on the job type and host name. This is applied on traces from Amazon EC2 running several scientific applications. The method reaches different performances on jobs from three different application settings, with FNs of 4%, 12% and 16% of total data points and corresponding FPs of 0%, 4% and 10% of total data points. This corresponds approximately to FPR of 0%, 5.7% and 16.3%, and TPR of 86.6%, 61.2% and 58.9%. The performance we obtained with our method is within a similar range for most benchmarks, although we never reach their best performance. However, we are predicting machine and not job failures.

A comparison of different classification tools for failure prediction in an IBM Blue Gene/L cluster is given in . In this work, they analyze Reliability, Availability and Serviceability (RAS) events using SVMs, neural networks, rule based classifiers and a custom nearest neighbor algorithm, trying to predict whether different event categories will appear. The custom nearest neighbor algorithm outperforms the others reaching 50% precision and 80% TPR. A similar analysis was also performed for a Blue Gene/Q cluster . The best performance was again by the nearest neighbor classifier (10% FPR, 20% TPR). They never evaluated the Random Forest or ensemble algorithms.

In an anomaly detection algorithm for cloud computing was introduced. It employs Principal Component Analysis and selects the most relevant principal components for each failure type. They find that higher order components are better correlated with errors. Using threshold on these principal components, they identify data points outside the normal range. They study four types of failures: CPU-related, memory-related, disk-related and network-related faults, in a controlled in-house system with fault injection and obtain very high performance, 91.4% TPR at 3.7% FPR. On a production trace (the same Google trace we are using) they predict task failures at 81.5% TPR and 27% FPR (at 5% FPR, TPR is down to about 40%). In our case, we studied the same trace but looking at machine failures as opposed to task failures, and obtained TRP values between 27% and 88% at 5% FPR.

All above-mentioned failure prediction studies concentrate on types of failures or systems different from ours and obtain variable results. In all cases, our predictions compare well with prior studies, with our best result being better than most.

# Discussion

Here we consider the possibility of including our predictive model in a data-driven autonomic controller for use in data centers. In such a scenario, both model building and model updating would happen on-line on data that is being streamed from various sources. From a technical point of view, on-line use would require a few changes to our model workflow.

To build the model, all features have to be computed on-line. Log data can be collected in a BigQuery table using the streaming API. As data is being streamed, features have to be computed at 5 minute intervals. Both basic and aggregated features (averages, standard deviations, coefficients of variation and correlations) have to be computed, but only for the last time window (previous time windows are already stored in a dedicated table). Basic features are straightforward to compute requiring negligible running time since they can be computed using accumulators as the events come in. For aggregated features, parallelization can be employed, since they are all independent. In our experiments, correlation computation was most time consuming, with an average time to compute one correlation feature over the longest time window taking 1489.2 seconds for all values over the 29 days (Table ). For computing a single value (for the newly streamed data), the time required should be on average under 0.2 seconds. This estimate corresponds to a linear dependence between the number of time windows and computation time and offers an upper bound for the time required. If this stage is performed in parallel on BigQuery, this would also be the average time to compute all 126 correlation features (each feature can be computed independently so speedup would be linear).

In terms of dollar costs, we expect figures similar to those during our tests — about 60 USD per day for storage and analysis. To this, the streaming costs would have to be added — currently 1 cent per 200MB. For our system, the original raw data is about 200GB for all 29 days, so this would translate to approximately 7GB of data streamed every day for a system of similar size, leading to a cost of about 35 cents per day. At all times, only the last 12 days of features need to be stored, which keeps data size relatively low. In our analysis, for all 29 days, the final feature table requires 295GB of BigQuery storage, so 12 days would amount to about 122GB of data.

When a new model has to be trained (e.g., once a day), all necessary features are already computed. One can use an infrastructure like the Google Compute Engine to train the model, which would eliminate the need to download the data and would allow for training of the individual classifiers of the ensemble in parallel. In our tests, the entire ensemble took under 9 hours to train, with each RF requiring at most 3 minutes. Again, since each classifier is independent, training all classifiers in parallel would take under 3 minutes as well (provided one can use as many CPUs as there are RFs — 420 in our study). Combining the classifiers takes a negligible amount of time.

All in all, we expect the entire process of updating the model to take under 5 minutes if full parallelization is used both for feature computation and training. Application of the model on new data requires a negligible amount of time once features are available. This makes the method very practical for on-line use. Here we have described a cloud computing scenario, however, given the relatively limited computation and storage resources that are required, we believe that more modest clusters can also be used for monitoring, model updating and prediction.

# Conclusions

We have presented a predictive study for failure of nodes in a Google cluster based on a published workload trace. Feature extraction from raw data was performed using BigQuery, the big data cloud platform from Google which enables SQL-like queries. A large number of features were generated and an ensemble classifier was trained on log data for 10 days and tested on the following non-overlapping day. The length of the trace allowed repeating this process 15 times producing 15 benchmark datasets, with the last day in each dataset being used for testing.

The BigQuery platform was extremely useful for obtaining the features from log data. Although limits were found for join and group by statements, these were circumvented by creating intermediate tables, which sometimes contained over 12TB of data. Even so, features were obtained with reduced running times, with overall cost for the entire analysis processing one month worth of logs, coming in at under 2000 USD[^2], resulting in a daily cost of just over 60 USD.

Classification performance varied from one benchmark to another, with Area-Under-the-ROC curve measure varying between 0.76 and 0.97 while Area-Under-the-Precision-Recall curve measure varying between 0.38 and 0.87. This corresponded to true positive rates in the range 27%-88% and precision between 50% and 72% at a false positive rate of 5%. In other words, this means that in the worst case, we were able to identify 27% of failures, while if a data point was classified as a failure, we could have 50% confidence that we were looking at a real failure. For the best case, we were able to identify almost 90% of failures and 72% of instances classified as failures corresponded to real failures. All this, at the cost of having false alarms 5% of the time.

Although not perfect, our predictions achieve good performance levels. Results could be improved by changing the subsampling procedure. Here, only a subset of the safe data was used due to the large number of data points in this class, and a random sample was extracted from this subset when training each classifier in the ensemble. However, one could subsample every time from the full set. However, this would require greater computational resources for training, since a single workstation cannot process 300 GB of data at a time. Training times could be reduced through parallelization, since the problem is embarrassingly parallel (each classifier in the ensemble can be trained independently from the others). These improvements will be pursued in the future. Introduction of additional features will also be explored, to take into account in a more explicit manner the interaction between machines. BigQuery will be used to obtain interactions between machines from the data, which will result in networks of nodes. Changes in the properties of these networks over time could provide important information on possible future failures.

The method presented here is very suitable for on-line use. A new model can be trained every day, using the last 12 days of logs. This is the scenario we simulated when we created the 15 test benchmarks. The model would be trained from 10 days of data and tested on one non-overlapping day, exactly like in the benchmarks (Figure ). Then, it would be applied for one day to predict future failures. The next day a new model would be obtained from new data. Each time, only the last 12 days of data would be used, rather than increasing the amount of training data. This to account for the fact that the system itself and the workload can change in time, so old data may not match current system behavior. This would ensure that the model is up to date with the current system state. The testing stage is required for live use for two reasons. First, part of the test data is used to build the ensemble (prediction-weighted voting). Secondly, the TPR and precision values on test data can help system administrators make decisions on the criticality of the predicted failure.

# Acknowledgments

BigQuery analysis was carried out through a generous Cloud Credits grant from Google. We are grateful to John Wilkes of Google for helpful discussions regarding the cluster trace data.

[^1]: Delfina Eberly, Director of Data Center Operations at Facebook, speaking on "Operations at Scale" at the *7x24 Exchange 2013 Fall Conference*.

[^2]: Based on current Google BigQuery pricing.