abstract: Candlestick charts display the high, low, opening, and closing prices in a specific period. Candlestick patterns emerge because human actions and reactions are patterned and continuously replicate. These patterns capture information on the candles. According to Thomas Bulkowski's Encyclopedia of Candlestick Charts, there are $103$ candlestick patterns. Traders use these patterns to determine when to enter and exit. Candlestick pattern classification approaches take the hard work out of visually identifying these patterns. To highlight its capabilities, we propose a two-steps approach to recognize candlestick patterns automatically. The first step uses the Gramian Angular Field (GAF) to encode the time series as different types of images. The second step uses the Convolutional Neural Network (CNN) with the GAF images to learn eight critical kinds of candlestick patterns. In this paper, we call the approach GAF-CNN. In the experiments, our approach can identify the eight types of candlestick patterns with $90.7\%$ average accuracy automatically in real-world data, outperforming the LSTM model.
author: Jun-Hao Chen
Department of Computer Science & Information Engineering
National Taiwan University
Taipei 10617, Taiwan
Yun-Cheng Tsai
School of Big Data Management
Soochow University
Taipei 11102, Taiwan
`email@example.com`
bibliography: template.bib
title: Encoding Candlesticks as Images for Pattern Classification Using Convolutional Neural Networks

# Introduction

Financial market forecasts are critical research topics in commercial finance and information engineering. For example, the topics are predicting fluctuations or volatility forecasts for futures indices . Market prices are susceptible to the expected psychological impact of the overall market. These prices are possible to develop predictive models of financial demand through particular pre-processing and complex model architectures.

Many tools are existing to help people predict stock price fluctuations and futures indices already . For example, these tools are the neural networks, fuzzy time-series analysis, genetic algorithms, classification trees, statistical regression models, and support vector machines. However, these machine learning models are generic techniques and used for forecasting. They unusually combine with financial expertise . Because the average person pursues profit in any transaction, the predictions of such models are not accurate enough for real-world operations. Investment forecasts and model predictions tend to have significant gaps, and investors are more inclined to find a good entry and exit point rather than merely predicting prices. Many studies focus on the accuracy of numerical predictions , but investors only concern with the time of entry and exit (i.e., how much profit space they have). In other words, rather than blindly using machine learning or deep learning architecture to pursue unrealistic low-risk, high-accuracy profit models, it is better to combine these directly with a basic knowledge of transactions to create a reliable, applicable model .

Candlestick pattern recognition is an essential tool for determining market conditions . To make trading decisions, traders often make judgments based on much-complicated information, such as technical indicators, news, and candlestick patterns. Thus, candlestick pattern recognition is a crucial support for individual transactions . Candlestick pattern recognition helps traders determine the current asset price in the market and establish whether the current buying pressure will continue or whether the current selling pressure will reverse. This information, along with other sources, assists traders to predict the future. Concerning price trends, the Morning Star and the Evening Star are examples of price reversal signals commonly. Candlestick pattern recognition requires a deliberate analysis of trader expertise rather than pure numerical analysis. This recognition requires traders to make visual judgments on images.

The Convolutional Neural Network (CNN) model is well-suited to image recognition . CNN can update its convolution kernel by backward propagation and train the appropriate weights to extract excellent image features. The correlation between traits and images uses to help models make correct judgments. Further, the type of neural network suitable for image identification needs to carry out through a two-dimensional convolution. Principally, the financial time-series data representing uses a one-dimensional array. Therefore, we need to find a way to convert the time-series data into a consistent matrix form.

However, our datasets are always dynamic, and patterns in them are changing. Hence, we need to feature engineering to extract specific time-series features. For example, space transformation models are kinds of feature engineering. There are including Singular Value Decomposition (SVD), distance metric learning, Nyström methods, and Distance Metric Learning (DML) approach . The process of Singular Value Decomposition (SVD) uses for investigation of the data. In these methods, linear algebra uses to construct a data matrix out of the collected data and to extract intrinsic features of that matrix. It is to separate elements that are similar between each subject and features that differentiate the items.

Instances with different labels are intertwined and often linearly inseparable. This issue brings new challenges to the CNN approach . The CNN approach considers unsuitable for directly encoding the time-series data as image pixels . Hence, we need a method for transforming time series data into images.

The Gramian Angular Field (GAF) has the following advantages:

1.  The GAF provides a way to preserve temporal dependency since time increases as the position moves from top-left to bottom-right.

2.  The GAF contains temporal correlations because the Gramian Angular represents the relative correlation by superposition and difference of directions for the time interval.

3.  The primary diagonal of the Gramian Angular Field matrix is the particular case.

4.  The diagonal of the Gramian Angular Field matrix contains the original value and angular information.

5.  From the main diagonal, we can reconstruct the time series from the high-level features learned by the deep neural network.

Hence, we use the Gramian Angular Field (GAF) to encode the time-series data  from a one-dimensional time-series array to the two-dimensional convolutional time-series matrix. The encoding data can improve the performance of the neural network in the two-dimensional convolutional time-series significantly. When the CNN model uses the GAF encoding as input, the LeNet  architecture can achieve outstanding results naively.

Therefore, we design a GAF-based CNN to emulate the trader to identify candlestick pattern characteristics in an experiment. We call our approach GAF-CNN. First, we use the Geometric Brownian Motion (GBM) model to simulate a volume of price data. According to Zhiguo, we set the same parameters to set the price, and its volatility is close to the real data . Second, we choose eight candlestick patterns from The Major Candlestick Signals . These eight types of pointers are Morning Star, Bullish Engulfing, Hammer, Shooting Star, Evening Star, Bearish Engulfing, Hanging Man, and Inverted Hammer. The difference between these eight candlesticks signals is subtle and will challenge a traditional CNN model.

To improve the traditional CNN model, we use the GAF-CNN to train the GBM simulation data. Our model produces outstanding performance in the simulation data. We also use real data to verify the viability of our GAF-CNN in the real-world. We expect that GAF-CNN enables the computer to look at the candlestick patterns with as much nuance as a trader. The results show a near-92% accuracy for the GBM simulation data. We use 2010-2017 historical data of the currency exchange rate for Euro (EUR) to US dollar (USD) to test our GAF-CNN model. The experimental results achieve a $90.70\%$ accuracy. The simulation and experimental results show that GAF-CNN is suitable for shape identification in financial trading. Although this paper uses only eight of the most classical-type indicators, various morphological extensions that can be made based on GAF-CNN are feasible, such as the W-head M-bottom. We want to establish a financial vision field through this paper making computers can recognize candlestick as a human has seen.

The remainder of this paper organizes as follows. Section 2 provides a review of the literature, and Section 3 presents our methodology. Section 4 shows the result of our experiments. Section 5 describes the discussion of Section 4. Section 6 is the conclusion of our study, and Section 7 is the overall workflow of our experimental framework.

# Preliminary

## Candlestick

Japanese start using technical analysis to trade rice in the 17th century . While this early version of technical analysis is different from the US version initiated by Charles Dow around 1900. Many of their guiding principles are similar. In this version, price action is more important than news and earnings. All happened information reflects in the price already. Buyers and sellers move markets based on expectations and emotions. The actual price may not reflect the underlying value. According to Steve Nison, candlestick charting first appears sometime after 1850 . Much of the credit for candlestick development and mapping goes to a legendary rice trader named Honma from the town of Sakata . His original ideas are likely modified and refined over many years of trading, eventually resulting in the system of candlestick charting used today.

Figure  is the structure of a candlestick. The unit is the bar, which draws on the opening, high, low, and closing prices (OHLC) for a specified period. The real-body is the price difference between the opening and closing prices. The upper shadow is the price difference between the highest price and the real-body, and the lower shadow is the price difference between the lowest price and the real-body. The period of a bar can be arbitrarily customized, usually depending on the length of the transaction. If the open price is higher than the close price, the real-body is rendered in black, indicating that the price is falling during this time. If the close price is higher than the open price, the real-body is white, indicating that the price is rising during this time. If the close price is equal to the opening price, the real-body will be just a (horizontal) line.

From the above, the candlestick helps investors filter out much of the price noise. The bar only records the different price information of OHLC per unit time. When we put together multiple bar charts, we get a continuous market information map. Unique shapes call as a pattern.

Researchers focus on the topic of candlesticks for many years . Many patterns use to identify trends summarized, such as trend continuation indicators or reversal indicators. Candlestick analysis is an approach to getting started with trading. However, some people think it is challenging to observe the trend by observing the candlestick. It cannot use as an indicator to predict direction . Human begins to systematize the patterns generated from the candlesticks. They evolve into technical indicators of the system to form the candlestick patterns gradually. The indicators are also including the Average True Range (ATR), Relative Strength Index (RSI), Moving Average (MA), Moving Average Convergence and Divergence (MACD), Stochastic Oscillator (KD)  and so on.

## Convolutional Neural Networks (CNN)

CNN models take advantage of the spatial properties of the data. According to Fukushima and Miyake, they propose a Neocognitron model. The model considers inspiring CNNs from the computational perspective generally . Neocognitron is a neural network designed to simulate the human visual cortex , which consists of two types of layers. The first type is the feature extractor layers, and the second type is the structured connection layers. The feature extractor layers, also named S-layers, simulate the cell in the primary visual cortex and help human beings to perform feature extraction. The structured connection layers, also named C-layers, affect the complex cell in the higher pathway of the visual cortex, provide the model with its shifted invariant property.

The two most essential components of CNN are the convolutional layer and the pooling (Pool) layer. Figure  shows that the convolutional layer implements the convolutional operation, which extracts image features by computing the inner product of an input image matrix and a kernel matrix. The number of channels of the input image and kernel matrix must be the same. For example, if the input image is a red-green-blue (RGB) color space, then the depth of the kernel matrix must be three; otherwise, the kernel matrix cannot capture the information between different color spaces. The pooling layer, also called the sub-sampling layer, is mainly in charge of simplifying the task. Figure  shows that the pooling layer only retains part of the data after the convolutional layer. It reduces the number of significant features extracted by the convolutional layer and refines the remaining features.

Only with these two components can the convolutional model be used to imitate human vision. In practical applications, the CNN model usually combines the convolutional layer and the pooling layer. The convolutional layer often extracts a significant number of features, and most of the elements may be noise, which could lead to the model learning in the wrong direction, also known as over-fitting. Furthermore, the fully-connected layers connect at the end of the sequence usually. The function of the fully-connected layer organizes the extracted features processed by the convolutional and pooling layers. The correlation between the extracted features learns in this layer.

Although the pooling layer can reduce the occurrence of over-fitting after convolution, it is inappropriate to use after the fully-connected layer. The other widely recognized regularization technique, called drop-out, designs to solve this issue. The drop-out technique randomly drops neurons with a specific probability, and the dropped neurons are not involved in the forwarding and backward computation. This idea directly limits the model's learning; the model can only update its parameters subject to the remaining neurons in each epoch.

The most general classic modern CNN model, LeNet inspires by Neocognitron and the concept of backpropagation . The potential of the modern convolution architecture can be seen in LeNet , consisting of a convolution layer, a subsampling layer, and a full connection (FC) layer . Figure  shows the LeNet model. As the concept of the rectified linear unit (ReLU) and drop out are presented in recent years, a new convolution-based model, AlexNet, proposed by Alex Krizhevsky and Hinton , appeared and beat the previous champion of the ImageNet Challenge, with 10M labeled high-resolution images and 10,000+ object categories.

## CNN for Patterns Classification

Human beings are visual creatures. The eyes are the most compact structure of all the sensory organs, and the visual intelligence of the human brain is rich in content. Exercise, behavior, and thinking activities all use visual sensory data as their most significant source of information. The more flexible and talented we become, the more we rely on visual intelligence. What general business and decision-makers desire after the analysis is not the data itself, but the value. Therefore, data analyses must be intuitive. In this way, the visualization of financial data more readily accept: decision-makers can see the story and interpret the data more efficiently.

Although visualization analysis can benefit decision-makers, many traditional statistical or machine learning methods for predicting currency movements use quantitative models. These methods do not consider visualization. We attempt to make good use of the advantages of display and comprehensively enhance the efficiency of intelligence analysis. For example, most traders use charts to analyze and predict currency movement trends, which carry apparent economic benefits. However, in this visualization, the analysis is artificial. We aim to teach machines to achieve the interpretation of visual information like a human brain. We then hope to use the tool to analyze robust financial data visually.

The CNN models use in pattern and image recognition problems widely. In these applications, the best possible accuracy has achieved using CNNs. For example, the CNN models have achieved a accuracy of 99.77% using the Modified National Institute of Standards and Technology (MNIST) database of handwritten digits , a accuracy of 97.47% with the New York University Object Recognition Benchmark (NORB) dataset of 3D objects, and a accuracy of 97.6% on over 5,600 images of more than ten objects. The CNN models not only give the best performance compared to other detection algorithms but also outperform humans in such cases as classifying objects into fine-grained categories, such as particular breeds of dogs or species of bird. The two main reasons for choosing a CNN model to predict currency movements are as follows:

1.  The CNN models are good at detecting patterns in images, such as lines. We expect that this property can use to detect trends in trading charts.

2.  The CNN models can detect relationships among images that humans cannot find easily. The structure of neural networks can help detect complicated relationships among features.

## Gramian Angular Field (GAF)

GAF is a novel time-series encoding method proposed by Wang and Oates , which represents time series data in a polar coordinate system and uses various operations to convert these angles into symmetry matrix. Gramian Angular Summation Field (GASF) is a kind of GAF using the cosine function. Each element of the GASF matrix is the cosine of the summation of angles.

Our first step to making a GAF matrix is to normalize the given time series data X into values between $[0, 1]$. The following equation shows the simple linear normalization method, where notation $\widetilde{x}_{i}$ represents the normalized data. $$\begin{aligned}
\widetilde{x}_{i}&=\frac{x_i-\min(X)}{\max(X)-\min(X)}
\end{aligned}$$ After normalization, our second step is to represent the normalized time series data in the polar coordinate system. The following two equations show how to get the angles and radius from the rescaled time series data. $$\begin{aligned}
\phi&=\arccos(\widetilde{x}_i), -1\leq \widetilde{x}_i \leq 1, \widetilde{x}_i \in \widetilde{X}\\
r&=\frac{t_i}{N}, t_i\in\mathbb{N}
\end{aligned}$$ Finally, we sum the angles and use the cosine function to make the GASF by the following equation: $$\begin{aligned}
\textup{GASF}=\cos(\phi_i + \phi_j) = \widetilde{X}^T \cdot \widetilde{X} - \sqrt{I-\widetilde{X}^2}^T\cdot \sqrt{I-\widetilde{X}^2}
\end{aligned}$$

The GASF has two essential properties. First, the mapping function from the normalized time series data to GASF is bijective when $\phi \in [0,\pi]$. In other words, normalize data to $[0, 1]$ can transform the GASF back into normalized time series data by the diagonal elements. Second, in contrast to Cartesian coordinates, the polar coordinates preserve absolute temporal relations.

# Methodology

This section begins with the overall experiment design, then illustrates the method of label creation, GAF-CNN model, feature selection, and neural architecture searching, respectively.

## Experiment Design

Considering real-world data lacking and complexity, it starts with simulation data to ensure GAF-CNN model work and progress feature selection and neural architecture search. Further, it will adopt in the empirical research on real-world data.

The simulation data are including the 2000 training data, 400 validation data, and 500 testing data from the Geometric Brownian Motion (GBM) model. Furthermore, we use EUR/USD 1-minute price data from January 1, 2010, to January 1, 2018, to label the real-world data, including 1000 training data, 200 validation data, and 350 testing data.

## Illustration of Label Creation

We select eight of the most classic candlestick patterns based on a classic candlestick patterns textbook, The Major Candlesticks Signals, as our training target. The eight candlestick patterns we chose are Morning Star, Bullish Engulfing, Hammer, Shooting Star, Evening Star, Bearish Engulfing, Hanging Man, and Inverted Hammer. All of these patterns are reversal patterns, which capture whether the price is going to change. The first four patterns detect the price from downtrend to uptrend, and the last four patterns detect the opposite. We illustrate Morning Star and Evening Star as examples below.

The Morning Star pattern detects a price changing from a downtrend to an uptrend. The description of this pattern has three stages. First, a downtrend must be confirmed, which means the whole market has an absence of confidence. Second, the depressed atmosphere results in a big black bar. After a calm day, the third bar is a big white bar, which indicates that the investors expect the confidence of the market to reverse. Figure  shows the main appearance and rules of Morning Star in detail.

The Evening Star pattern detects the price changing from an uptrend to a downtrend. The description of this pattern also has three stages. First, an uptrend must be confirmed, which means the whole market is in a specific situation. Second, good days end with a big white bar. After a calm day, the third bar becomes a big black bar. These indicate that the investors expect the confidence of the market to reverse. Figure  shows the main appearance and rules of Evening Star in detail, and Figure  shows the difference between Morning Star and Evening Star patterns.

The definition of our label bases on the rules given in The Major Candlesticks Signals, as shown in Figures  and . The downtrend and uptrend define from regression. If the slope is higher or lower enough, the trend is confirmed. The definition of slope in our implementation is as follows, Figure  has the entire illustration:

1.  The slope value computes from the closing price among 7 bars.

2.  Move a bar window to get another slope value.

3.  Keep collecting positive and negative slope until 50 units, respectively.

4.  If the current slope is over the 70th percentile of the group, then it will be defined as a positive or negative trend.

We must note that the other pattern rules are slightly different between the simulation and the real data. The rules from the simulation data are similar to the book. Nevertheless, the number of samples is insufficient in real-world data because of the strictness of the rules. Hence, we relax the rules to obtain sufficient data slightly. For example, the Bullish Engulfing pattern requires the opening price of the last bar to be lower than the closing price of the previous bar. If this rule is too strict, we relax the condition such that the opening price of the last bar only needs to be less than or equal to half of the real body of the previous bar.

## GAF-CNN

We propose a two-step approach and call it the GAF-CNN model. The first set is the Gramian Angular Summation Field (GASF) time-series encoding, and the second step is the Convolutional Neural Networks (CNN) model. In the first step, we encode time series data based on opening, high, low, and closing prices (OHLC) to GASF matrices with the window size set to $10$. After this step, the shape of the data matrices will be $(10, 10, 4)$. In the second step, we train this 3-d matrices data with the CNN model. The architecture of our second step's CNN model is similar to LeNet, including two convolutional layers with $16$ kernels and one fully-connected layer with $128$ dense. Figure  illustrates the entire experimental architecture, and Table  shows the parameters used in our GAF-CNN model.

|   Parameters   |             |
|:--------------:|:-----------:|
|     epochs     |    $300$    |
|   batch size   |    $64$     |
|   optimizer    |    Adam     |
| learning rate  |   $0.001$   |
|     beta 1     |    $0.9$    |
|     beta 2     |   $0.999$   |
| early stopping | $20$ epochs |

The parameters of our GAF-CNN model.

## Features Selection

According to the previous section, the candlestick patterns cannot judge from a single value such as closing or opening price. Therefore, we need to combine opening, high, low, and closing prices (OHLC) and make the data features more reasonable. In order to close to humans have seen, we consider using the upper shadow, lower shadow, and real-body, which are more intuitive features for humans. Figures  and  are based on different features respectively of the Morning Star and Bearish Engulfing patterns through

1.  the opening, high, low, and closing prices (OHLC); and

2.  the closing price, upper shadow, lower shadow, and real-body (CULR).

Figures  and  show the visualization of the GASF matrix in two kinds transformation rules. Figure  shows more capable of extracting distinctive features observed than figure . Because the differences between the opening, high, low, and closing prices (OHLC) are generally small, resulting in high similarity among these four GASF matrices. If the model has too much repetitive information, this repeat information will reduce the convolutional model's effectiveness in learning critical features. Hence, we process the data into the features of the second transformation rule (CULR). When we use this transformation rule, the four features are not similar and pop out the significant 2-D features in the GASF matrix. From another perspective, this is a more intuitive approach that aligns with the observations of traders. Therefore, we design our experiments using

1.  the opening, high, low, closing prices (OHLC); and

2.  the closing prices, upper shadow, lower shadow, real-body (CYLR) features

in the simulation data. The better results are later applied to the real-world data.

## Neural Architecture Searching

The GAF-CNN model works well with the simple neural architecture, two convolutional layers with $16$ kernels, and one fully-connected layer with $128$ denses. The max-pooling layer, which uses general picture classification, calculates the maximum value for each patch of the feature map usually. In other words, it may bring benefits about calculating cost-saving, but truncate the characteristics of the time series, which means discard information of data. Therefore, we design an experiment using a max-pooling layer or not in simulation data. Figure  illustrates where to use the max-pooling or not.

# Results

## Baseline

Previous research on the candlestick with deep learning is about trading strategy but lack of pattern classification. It is hard to find the result from other studies to compare the GAF-CNN model, so we chose the Long Short-Term Memory model (LSTM) for reliable comparison since it is a standard method to accomplish the time series classification or regression tasks in the current year. Our goal is to achieve or surpass the performance of the LSTM model. The architecture used in this study include two hidden layer size of 128 LSTM layer and follow by a 128 dense layer . More detail comparisons will discuss in sections 4-2 and 4-3.

## Simulation Results

Figure  shows the result comparing between different features and neural architectures mention in section 3. Each experiment searches $100$ times to find out the best model and predict testing data.

The GAF-CNN model without the max-pooling layer can achieve $92.42\%$ accuracy, which is better than the LSTM model $88.96\%$ accuracy in both feature sets. Figures  and  respectively show the confusion matrix of GAF-CNN model without max-pooling layer and with the different feature sets:

1.  the opening, high, low, closing prices (OHLC); and

2.  the closing prices, upper shadow, lower shadow, real-body (CULR).

The result of using (2) closing, upper shadow, lower shadow, and real-body (CULR) can achieve $92.42\%$ average accuracy. If we focus on the result from class 1 to class 8, then the performance is $95.43\%$ accuracy on average.

Figures  and  show the confusion matrix of LSTM model with two feature sets respectively. The accuracy of using (1) opening, high, low, closing prices (OHLC) is $88.58\%$ on average, and using (2) closing, upper shadow, lower shadow, real-body (CULR) is $88.96\%$ on average.

To explore more about the model training process, a comparison of the first 50 epochs under different conditions would help to realize the rate of convergence. Figure  and Figure  depict the difference of both feature sets and using max-pooling or not respectively.

## Empirical Results

EUR/USD 1-minute price data from January 1, 2010, to January 1, 2018, are used in our real data framework, including 1000 training data, 200 validation data, and 350 testing data. Therefore, we used two times as much data in training set for class 0, which is the noisy data for the other classes. The purpose of this is to help the model clearly distinguish the patterns and increase the robustness.

Based on the results of the simulation data, we chose to use closing, upper shadow, lower shadow, and real-body (CULR) as our feature set, and to exclude the pooling layers in our model. Figure  shows the confusion matrix of the real-world framework. The GAF-CNN model achieves $90.7\%$ accuracy on average in real-world data.

# Discussion

## Simulation Results

First of all, Figure  shows that using (2) closing prices, upper shadow, lower shadow, a real-body (CULR) feature set can significantly improve the accuracy in the GAF-CNN model than using (1) the opening, high, low, closing prices (OHLC) feature set. In Figure , the training process also converges significantly faster in the first 50 epochs, and end up with higher accuracy. This result is intuitive that this feature set is more close to trader way, observing the characteristics of the candlestick.

Secondly, the model also converges faster when using (2) without max-pooling layer than (1) with the max-pooling layer in Figure . In Figure , the GAF-CNN model without the max-pooling layer can achieve higher accuracy and lower loss value in both feature sets. The result can explain that the dependency on time series data contains many essential features. The complete time-series information will be truncated after the processing of the max-pooling layer, making it harder for the convolutional model to capture more detail features.

Lastly, the GAF-CNN model works well in both simulation and real-world framework. It achieves $90.7\%$ accuracy on average in real-world data. Besides, our results show that class 0, which is the other class, has reduced precision and recall. The class does not affect the usability of the framework because, although class 0 does not perform well, as long as the accuracy of the other classes is high enough, the cost of misclassification is small.

## Empirical Results

The result in Figure  shows that GAF-CNN can achieve $90.7\%$ on average in the real-world data, outperforming the result of LSTM model. Therefore, our experimental results show that the GAF and the CNN framework are well-suited for candlestick pattern recognition for both simulation and real-world trading data.

# Conclusions

Candlestick pattern recognition is an indicator that traders often judge with news, fundamentals, and technical indicators. However, even today, most traders decide by using their vision and experience. Although many people have directly drawn up rules to find patterns, the process is too cumbersome and hard to judge without the provision of soft scores. To better align with how traders identify patterns, we chose to use the two-dimensional CNN model. We used the GAF time series encoding with the traditional CNN model Because of the direct use of images to train leads to underfit. We use GAF-CNN to process the GBM simulation and EUR/USD real word experiments.

In the simulation framework, we use eight candlestick patterns to test how the max-pooling layer and feature sets impact our model. The results indicate the following:

1.  The max-pooling layer is terrible for the GAF-CNN model. We think that the time series are truncated and lead to the loss of practical information.

2.  Using the feature set of closing price, upper shadow, lower shadow, and real-body (CULR) is better than using the simple feature set of opening, high, low, and closing prices (OHLC).

The model achieved an average accuracy of $92.42\%$ in simulation data. Although the 0 class is prone to misclassification, the model is still available for practical work as long as the main pattern resolutions and recall are high enough.

In the real-world framework, we use the same model for the EUR/USD per minute data from January 1, 2010, to January 1, 2018 retraining, including 1000 training data, 200 validation data, and 350 testing data. The model obtained $90.7\%$ average accuracy, outperforming the LSTM model. In real-world data, class 0 has more false positives than other types, but the main kind of recall is a certain extent. It can be considered a more conservative model. Finally, because the difference between these eight indicators is tiny, GAF-CNN has to extract subtle features. Now we only use the eight main candlestick patterns. Furthermore, future work could apply GAF-CNN to more candlestick patterns or technical indicators, such as W-head M-bottom. Thus, the entire architecture in finance candlestick, and the extensibility of the models is enormous.

# Workflows

In this study, we find that the Convolutional Neural Network model can detect financial time series data effectively, and our research workflow is as follows:

1.  Our experiments adopt simulation, and real-world framework, where the simulation data generates from Geometric Brownian Motion model and the real data is EUR/USD per minute data from January 1, 2010, to January 1, 2018.

2.  Eight candlestick labels reference from The Major Candlestick Signals.

3.  Use opening, high, low, and closing prices (OHLC) or closing, upper shadow, lower shadow, and real-body (CULR) feature sets. The data in this stage is still a 10 by 4 matrix, where 4 represents the features.

4.  Encode time series data by Gramian Angular Summation Field. The data will become 10 by 10 by 4 in this stage.

5.  Each framework of training, validation, and testing is with the Convolutional Neural Network model.

The first step is each experiment test in the simulation framework, then apply the result of feature sets and neural architectures to the real-world framework. In all experiments, the convolution model use only two convolutional layers with 16 kernels and one fully-connected layer with 128 denses. All these processes illustrate in Figure .

# Abbreviations

# Declarations

## Availability of data and material

We provide an open source (https://github.com/RainBoltz/Series2GAF) Series2GAF which can be used to transform time series into Gramian Angular Field.

## Funding

Jun-Hao Chen and Yun-Cheng Tsai are supported in part by the Ministry of Science and Technology of Taiwan under grant 108-2218-E-002-050-.

## Authors contributions

Yun-Cheng Tsai conceived of the presented idea. Jun-Hao Chen developed the theory and performed the computations. Yun-Cheng Tsai and Jun-Hao Chen verified the analytical methods. All authors discussed the results and contributed to the final manuscript.

## Acknowledgements

Thanks to Prof. Jane Yung-Jen Hsu for constructive discussion and great support.

# Competing interests

Jun-Hao Chen and Yun-Cheng Tsai declare that we have no significant competing financial, professional or personal interests that might have influenced the performance or presentation of the work described in this manuscript.


abstract: ViZDoom is a robust, first-person shooter reinforcement learning environment, characterized by a significant degree of latent state information. In this paper, double-Q learning and prioritized experience replay methods are tested under a certain ViZDoom combat scenario using a competitive deep recurrent Q-network (DRQN) architecture. In addition, an ensembling technique known as snapshot ensembling is employed using a specific annealed learning rate to observe differences in ensembling efficacy under these two methods. Annealed learning rates are important in general to the training of deep neural network models, as they shake up the status-quo and counter a model's tending towards local optima. While both variants show performance exceeding those of built-in AI agents of the game, the known stabilizing effects of double-Q learning are illustrated, and priority experience replay is again validated in its usefulness by showing immediate results early on in agent development, with the caveat that value overestimation is accelerated in this case. In addition, some unique behaviors are observed to develop for priority experience replay (PER) and double-Q (DDQ) variants, and snapshot ensembling of both PER and DDQ proves a valuable method for improving performance of the ViZDoom Marine.
author: Christopher Schulze, Marcus Schulze
{cschulze7, firstname.lastname@example.com [^1]
title:  **ViZDoom: DRQN with Prioritized Experience Replay, Double-Q Learning, & Snapshot Ensembling\***

# INTRODUCTION

Increasingly, deep reinforcement learning (DRL) is the topic of great discussion in the artificial intelligence community . For reinforcement learning (RL) experts, RL has always shown great promise as a robust construct for solving task oriented problems. Recently, the DRL variant has gained significant public attention. But to be clear, much remains to be done at large in the field concerning high state-action dimensional problems that approximate well some real world problems of interest. From board games like Go in recent engagements with agents from Deep Mind to very high state-action dimensional games like those from the real time strategy (RTS) and first person shooter (FPS) video game genres, DRL has established itself, presently, as a prime construct for solving highly complex problems without direct supervision for most, if not all, of its training.

Given the success, much work has been done recently to improve upon models of DRL, including that of the Deep Q-Network (DQN). Priority experience replay (PER) and Double-Q learning (DDQ) are two such methods that can be used to improve a DQN agent's rate of improvement or degree of learning stability, respectively, during training. However, to the authors knowledge, these two methods have yet to be tested under the ViZDoom \[8\] environment - a setting characterized by a higher degree of latent information relative to Atari and other popular RL environments - in the context of a specific, effective deep recurrent learning architecture. Replicating the DRQN structure in the paper from Lample et al.\[5\], the authors in this paper test the benefits offered by PER and DDQ methods under an efficient ensembling method.

# DEEP REINFORCEMENT LEARNING OVERVIEW

## Reinforcement Learning, Q-Learning, and Deep Q-Networks

The field of reinforcement learning frames a learning problems using two separate entities: an agent and an environment. As Sutton discusses \[10\], it is useful to conceptualize this framework as follows: an agent is defined as an actor attempting to learn a given task. The agent is delineated from the environment by defining its characteristics and action set as items encompassed by its realm of influence and under its complete control; all other aspect of the learning problem are then attributed to the environment. Generally, the environment can be described as the setting in which the learning problem takes place. The agent receives its current state $s_{t}$ from the environment, and it proceeds to interact and affect the environment through action $a_{t}$ , which is derived from a policy $\pi_{t}$ . The environment takes in action $a_{t}$ and updates the current state of the agent to $s_{t+1}$ along with sending a reward signal $r_{t}$ to the agent, indicating the value of $s_{t}$ to $s_{t+1}$ transition via action $a_{t}$. Given the RL framework, we can define discounted rewards at time $i=0$ as :

$$R = \sum_{i = 0}^{T}\gamma^i r_i \eqno{(1)}$$

, where $\gamma$ $\in$ \[0,1). Rewards are discounted to simulate the concept of delayed rewards to the agent, the importance of the discounted being directed by the value of $\gamma$. $\gamma$ values close to 0 indicate immediate rewards are more important; whereas $\gamma$ values close to 1 indicate to the agent that longer of sequences of actions are important to consider to achieve high rewards. The goal of the agent is to maximize its rewards as seen by the environment over the course of a single experience or series of experiences (i.e games) by developing a policy. This policy dictates what action an agent performs given its current state.

To obtain an approximation of optimal values corresponding to state-action pairs, the temporal difference method \[11\] Q-learning, first developed by Watkins \[3\] in 1989, can be used. The Q-function for a given state-action pair with policy $\pi$ is as follows,

$$Q^{\pi}(s,a) = E[R_i | s_i = s, a_i = a] \eqno{(2)}$$

The task is then to find the optimal policy, giving:

$$Q^{*}(s,a) = max_{\pi}(Q^{\pi}) = max_{\pi}(E[R_i | s_i = s, a_i = a]) \eqno{(3)}$$

Thus, Bellman optimality is then satisfied by the optimal Q-function, as the above can be rewritten as:

$$Q^{*}(s,a) = max_{\pi}(Q^{\pi}) = E[r + \gamma * max_{a'}( Q^{*}(s',a') )  | s, a ] \eqno{(4)}$$

For most problems - with large state-action spaces - direct, recursive methods, including dynamic programming methods, are impractical. Rather, the optimal Q-function is approximated by a parametrized method. In particular, deep learning architectures have proven very successful at approximating the optimal value of high-dimensional target spaces. Let $Q_w$ be defined as a Q-function with parameters w. For updates to the $Q_w$ function, the loss can be defined as:

$$L_{t}(w_{t}) = E_{s,a,r,s'}[ l_{t}(y_{t} -  Q_{w}(s,a)) ] \eqno{(5)}$$

, where $l_{t}$ is any reasonable transformation of the temporal difference error for training deep neural networks, including L1 (proportional to mean absolute value error) and L2 (proportional to mean squared error) losses, among others, and

$$y_{t} = r + \gamma * max_{a}(Q_{w}(s',a))$$

Given successes with stochastic gradient descent updates, we can instead lose the expectation and give stochastic updates to $Q_{w}$ using the following as loss for backpropagation:

$$L_{t}(w_{t}) = l_{t}(y_{t} -  Q_{w}(s,a)) \eqno{(6)}$$

To gain experiences in the environment, $\epsilon$-greedy training can be used, wherein the agent randomly acts at with probability $\epsilon$ or choses what it deems its best action with probability 1 - $\epsilon$. Using this strategy, $\epsilon$ is decayed over the course of training, usually starting with a value of 1 and having a minimum value of 0.1.

To stabilize Q-values during learning, a replay memory is used to remember $s,a,r,s'$ experiences as the agent interacts with the environment; the agent then learns by sampling from this replay memory uniformly after a set number of games played and fitting against the selected experiences. The replay memory along with a target Q-function were introduced to counter the algorithm's strong affinity to local optima given these greedy updates.

Enter the modern framework for Deep Q-Networks in full. Deep Q-Networks (DQNs) are a a parametrized, model-free, off-policy method; specifically, it uses Q-learning for value estimation. Two deep neural network architectures are used to learn the Q-values for each experience sampled from replay memory. An online network $Q_w$ learns in a greedy fashion, and the target network $\tilde{Q}_w$ acts as a tether point to reduce the likelihood of the online network from falling into a local optima due to its greedy updates.

Let $Q(s,a)$ be defined as the online network, and $\tilde{Q}(s,a)$ be defined as the target network. During training using replay memory after a number of games have been played, the online network is updated using the following target:

$$Q_{target}(s,a) = r + \gamma max_{a}(\tilde{Q}(s',a)) \eqno{(7)}$$

## Deep Recurrent Q-Networks

A variant of DQN, deep recurrent Q-Networks (DRQNs) have shown exceptional performance in a variety of RL problems. In environments characterized by significant latent information, recurrency offers a way to fill in the gaps of missing knowledge for a given state\[7\]. The RL environment of ViZDoom is no exception; as a first person shooter, the agent is bound by a first-person, 90-degree view of objects in front of it. There is no radar - a HUD display of enemies around the player in a 360 degree arc is present in many other FPS games - or other indicators of what could be in the other 270 degrees.

As such, rather than receiving states $s_t$ as in the RL framework, it is more aptly put that the agent in ViZDoom receives partial observations $o_t$ of its current state $s_t$ \[7\]. Thus, using a DQN, the objective is to estimate not $Q(s_t, a_t)$ but instead $Q(o_t, a_t)$, putting the agent at a distinct disadvantage concerning state level information. One way to counter this is to include state information from the previous sequence of states, thereby allowing the agent to instead estimate $Q(o_t, h_{t-1}, a_t)$ where $h_{t-1}$ represents information gathered at state $t-1$ and passed to state $t$. The long-term, short-term memory cell, LSTM, is one recurrent construct capable of doing this; at a given time $t$, the LSTM cell takes in $o_t$ and $h_{t-1}$ and outputs $h_t$. Instead of $Q(o_t, a_t)$, the network then estimates $Q(h_t, a_t)$, increasing the level of information available to the agent for a given state-action pair.

## Double-Q Learning

Double-Q learning advances upon DQN in a simple, yet remarkably effective way: let the loss used in learning by the agent be defined by the value-maximizing actions of the online network with the Q-values of the target network associated with those maximizing actions. With the decoupling of the maximizing action from its value, it is possible to eliminate the maximization bias present in Q-Learning \[6\]. Specifically, the loss is defined as follows:

$$L_t(w_t) = l_t(y_t -  Q_{w}(s,a)) \eqno{(8)}$$

,where $l_t$ is a transformation of the temporal difference (TD) error and

$$y_t = r + \gamma \tilde{Q}_{w}(s', a_{online}))$$

$$a_{online} = argmax_{a}(Q(s', a))$$

This innovation was spurred by the observation that Q-learning tends to overestimate the values of state-actions pairs. Through experimentation of deep double-Q learning using myriad, diverse Atari game environments, Hasselt et. al. find that deep double Q-networks(DDQNs) show marked improvement in the estimation of values for state-action pairs \[14\]. Furthermore, Double-Q learning introduces a level of stability in Q-learning updates, which allows for more complex behavior to be learned.

## Prioritized Experience Replay

During training, the DQN agent samples uniformly from its replay memory to populate a batch training set and subsequently learn on this set. This is done many times over the course of the experiment, allowing the agent to continue to learn from previous experiences. The construct of replay memory was created to simulate learning from experiences that are sampled i.i.d (independently and identically distributed). Without approximating i.i.d sampling, the agent can quickly overfit to recent state-action pairs, inhibiting learning and adaptation. The method of prioritized experience replay innovates on this front by biasing the sampling of experiences \[12\]. Specifically, experiences are weighted and sampled with higher probability according to the TD error observed for that sample - the larger the TD error, the higher the probability with which a given experience will be sampled. The intuition behind this is that experiences that significantly differ from the agent's expectation will have more didactic potential. Concerning empirical validation of the method, Schaul et al. created PER and showed that it affords significant performance improvement across many Atari-based RL environments. The authors here apply PER to a DRQN model and train and test it in the defend-the-center ViZDoom scenario (see IV. A. Scenario section).

## Ensembles of Deep Q-Networks - Snapshot Ensembles

Ensembling of models has proven useful in many situations, countering, to an extent, the tendency of nonlinear models to over-fit to a given training distribution. However, the generation of ensembles for deep neural networks can prove onerous, requiring the training of multiple networks in parallel or - even worse, in terms of total training time - sequentially. Recently, methods have been proposed to gain the advantages of ensembling while reducing the time to generate such a set of models. Snapshot ensembling is one such example, employing a cosine annealing learning rate to generate $M$ models over $T$ total epochs from the training of a single model over those $T$ epochs \[4\]. This is done by using the cosine annealing learning rate to train the single model and take "snapshots" of its current weights every $T/M$ epochs. Thus, only a single model is trained while, at the same time, providing a diverse model population for the ensemble through use of the cosine annealing learning rate.

The authors here use the snapshot ensemble method to analyze performance improvement of the aforementioned model (DRQN) in the context of the ViZDoom Reinforcement Learning Environment, utilizing the learning enhancement methods of PER and DDQ.

## Review of Modern RL Game Environments

## For Reinforcement Learning, Why Video Games?

A burgeoning field of work has been created using the VizDoom environment due to the unique problems that the 3-D first-person shooter (FPS) engine can provide. Navigation, recognition of objects in the space, and decision making when other objects or actors are encountered are all obstacles that an FPS environment presents. Another important obstacle is that the agent will never have a complete view of the given state space. Recent deep reinforcement learning methods applied to this environment have focused primarily on the free-for-all death match setting to train their models. Instead, the authors here choose to train and test models using the Defend the Center mode.

Including ViZDoom, there are a number of diverse game-based RL frameworks in current use. In general, video games have increasingly become an important tool for AI research and development. It is easy to see why, as video games allow for an environment rich with parameters, feedback, and end goals for agents to gauge their success. Originally, the frameworks tested were simple 2-D environments from games originally on the Atari 2600 that allowed for simple movement and near fully observable states. On the other hand, the engines for FPS environments provide a wealth of data and multiple obstacles and objectives for the AI agents to learn from. One of the most important obstacles provided by the FPS setup is the lack of complete information for a given state. This obstacle mimics real world problems for autonomous agents, as they must be able to act with a limited view of the world around them.

## Arcade Learning Environments

The classic 2D video games of the past have been used to a great extent for deep learning. The main platforms for these games are the Atari 2600, Nintendo NES, Commodore 64 and ZX Spectrum. One of the most used emulator environments is Stella, which uses the Atari 2600 and has 50 games available. Methods previously used in these environments include Double DQN, Bootstrapped DQN, Dueling DQN, and Prioritized DQN. Montezuma's Revenge is a notable game from this genre as it requires memorization of other rooms that aren't immediately available to the agent. Methods used in this space are DQN-PixelCNN, DQN-CTS, and H-DQN \[2, 9\].

## Real Time Strategy

StarCraft: Brood War is a popular real time strategy (RTS) game in which the objective is to destroy all of the enemy structures and claim victory. Players move towards victory by gathering resources to create attacking units that can then be directed in various actions. The obstacles for the agent are many as the state space is complex and not fully observable by the agent at any one time. There are three factions available in this environment that all have their own unique characters and abilities. Even if the agent is limited to learning to play only a single faction, there are still 3 different match ups that could be encountered. Each of these match ups will have their own strategies and units that will be needed to counter different compositions of units built from different structures.

Another important skill that the agent needs to learn is intelligence gathering - understanding of both map layout and composition of the opponent's army. At the start of the game, the map is under what is called "fog of war", which blacks out any area that doesn't have a controlled unit to provide vision. In summary, the agent must navigate the environment and gather intelligence on the opposing player, build an appropriate base from which to train units to defend and attack the opponent, maneuver these units into advantageous positions in the environment, engage opponent units and manage the abilities available to units, and manage resources. All of these tasks must also be performed without complete knowledge of the state space. The combination of the many problems experienced in the course of one game has led deep learning researchers to focus on specific problems in the game as the sparsity of rewards makes the training of highly non-linear functions, such as neural networks, difficult. The main problem that has been the focus of many researchers has been the micromanagement of units in combat scenarios. Methods such as IQL, COMA, Zero Order, and BiCNet have been performed with promising results \[9\].

## Open World Games

Open world games are another avenue of research, as the nature of open world games positions issues of exploration and objective setting as the main challenges to the agent. Project Malmo is an overlay built onto the Minecraft engine that can be used to define the environment and allow for objectives to be imposed on an agent in a normally free task environment. These large and open problems are commonly used to test reinforcement learning methods. Methods such as H-DLRN and variations of NTMs have been successful in preforming varying tasks in the space \[9\].

## Racing Games

Racing games are another popular genre for AI research as there are many challenges here as well. Depending on the game chosen, the inputs for control can be as complex as having a gear stick, clutch and handbrakes, while others are much more simplified. Challenges for this environment include positioning of the vehicle on the course for optimal distance traveled, adversarial actions to block or impede other drivers when other vehicles are present, and sometimes the management of resources. This genre is also useful because the entire state is not available to the agent in the first or third person view. Methods that have been used in this genre include Direct Perception, Deep DPG and A3C, with a popular environment for this genre being the simulator TORCS \[9\].

## First-Person Shooter

VizDoom is based off of the popular video game Doom and is a first person shooter. In this environment there are many challenges that make it a useful tool for training AI. There are many modes of play within the VizDoom environment, two of which are mentioned here: Defend the Center and Deathmatch. In Defend the Center, the agent is spawned in the center of a circular room and is limited to turning left or right to find and eliminate the enemies that spawn. At most, only 5 melee enemies are present on the map at any one time. These are spawned against the wall; they then make their way towards the agent in the center. As enemies are eliminated, others are spawned to take their place. The objective is to hold out for as long as possible with a limited amount of ammunition. Having limited ammo helps to constrain episodes to a limited time limit, as when ammo is depleted, defeat is inevitable. In Deathmatch the objective is to reach a certain amount of frags (i.e. kills) before other actors on the map or to have the most frags when time runs out. The view of the agent is determined by the resolution of the screen chosen and varies between 90 and 110 degrees, meaning that the agent doesn't have complete knowledge of its full state at any time. Other obstacles include the recognition of enemies in the space, aiming weapons, and navigation of space. Methods that have been used in this environment include DQN+SLAM, DFP, DRQN+Auxiliary Learning, and A3C+Curriculum Learning \[8, 9\].

# MODEL ARCHITECTURE AND AGENT TRAINING

## Deep Neural Network Architecture

The authors here use the DRQN model architecture (Figure 2) specified by Lample et al. \[5\]. This model architecture was used in all experiments, as it was noteworthy in its performance without the additions of PER and DDQ; thus the authors here endeavored to use PER and DDQ to observe how such variants can aid a deep learning architecture that is well suited to this RL problem. A general description of the architecture is as follows:

-   input to the network is a single frame, i.e. the 3 channels (RGB) of a frame, with each frame being resized to 60x108 pixels

-   input is sent through two convolutional layers

-   the output of the final convolutional layer $C_f$ is flattened and sent to two destinations

-   output of $C_f$ is sent to a dense layer and then to a subsequent dense layer $E_d$ with sigmoid activation of size 1. This is sent to a recurrent layer $R_l$ and is also used for loss

-   output of $C_f$ is flattened and then sent directly to the recurrent layer $R_l$

-   $R_l$ then outputs to a dense layer with ReLu activation and then to a subsequent dense layer with linear activation of size three, as there are three actions that can be performed at a given time (turn left, turn right, and shoot) in the Defend-the-Center scenario.

The size-1 dense layer is a boolean indicator for enemy detection and is fitted by querying the game engine for enemies in the agent's vision; if there are any enemies in the agent's view, the true value is 1, otherwise it is 0. The feeding of predicted enemy detection information into the recurrent layer was found by Lample et al. to be very beneficial in training in a related ViZDoom RL scenario known as Deathmatch, where agents engage in a competitive, FPS match.

## Frame Skipping, Fitting, and Reward Structure

Frame skipping is quite crucial in the training of RL agents that use video data with a reasonably fast frames-per-second (fps) value \[10\]. For example, using a frame-skip value of 2, the agent will make an action on frame 0, $f_0$. That same action will be performed for the skipped frames, $f_1$ and $f_2$. There are many reasons for using frame-skipping, one of the main benefits being that it prevents the agent from populating the replay memory with experiences that differ almost imperceptibly from one another. Without frame-skipping, this can pose a serious issue for training DRQN's, as the elements of sequences that are sampled for training will be practically the same, often causing the agent to learn degenerate, loop-like behavior where it performs a single action over and over. For all experiments, the ViZDoom engine was run at 35 fps. The authors here experimented initially with various frame skip values, opting for a frame-skip value of approximately 10 for reported results.

For training of a DRQN, sequential updates are important in order to take advantage of the recurrent layer. Here, the authors sampled sequences of length 7, used the first four experiences as primer for training - passing LSTM cell state information sequentially from the previous experience to the next - and trained on the final 3 experiences.

Agents were trained for a total of 11500 games in the defend-the-center scenario, using $\epsilon$-greedy training. Here, $\epsilon$ was decayed from 1.0 to 0.1 over the course of training. For the three experiments without snapshot ensembling, a linearly decaying, cosine annealed (small amplitude) learning rate was used. For the other three using snapshot ensembling, a cosine annealed learning rate was used. Instead of using Adam, which combines classical momentum with the norm-based benefits of RMSProp, all models were optimized using the Nesterov Adam optimizer, as this swaps out classical momentum for an improved momentum formulation, Nesterov's accelerated gradient (NAG) \[13\]. In terms of reward structure, the agent was given a reward during training of $+1$ for frags of enemy units, and a penalty of $-0.5$ for deaths. A penalty proportional to the amount of health lost was also included.

## Hyperparameters

Hyperparameter values for all experiments are as follows:

-   $\gamma$ value of 0.9

-   for PER, a $\beta_{0}$ value of 0.5 was used, and $\beta$ was increased to 1.0 linearly over the course of each experiment

-   for PER, $\alpha$ value of 0.7

-   for PER, $\epsilon$ offset of 0.05 for non-singular priority values

-   a batch size of 20 with sequence length 7 was used for replay memory training

# EXPERIMENTS

## Scenario

Previous literature concerning ViZDoom has focused on using the Deathmatch scenario to train their agents in navigation, combat, and enemy recognition. The authors here focus on the problem of spatial recognition as opposed to navigation. In order to emphasize this, the game mode Defend the Center available in the Vizdoom engine was chosen. This goal of this game type is to frag as many adversaries as possible before you are overrun. The agent is allowed no movement in the center of this circular arena other than the adjustment of its angle of view. With this limitation, the authors here aim to have agents learn spatial awareness and to prioritize the targets based on distance from the agent. As a review of Section II F, in the Defend the Center scenario, the agent is spawned in the center of a circular room and is limited to altering the degree of its view to find and eliminate the enemies that spawn. At most only 5 melee enemies are spawned against the wall (Figure 3) that will then make their way towards the agent in the center. The objective is to hold out for as long as possible with a limited amount of ammunition.

## Software and Hardware

Six experiments in total were performed, two sets of 3 experiments each. The first set was to test the base DRQN and PER and DDQ variants without snapshot ensembling, while the second set employed the use of snapshot ensembling. In both cases, the authors here use the proportional PER version, rather than rank PER. In all experiments, agents were trained for approximately 12 hours for a total of 11500 games of the ViZDoom defend-the-center scenario on an NVIDIA GeForce Titan X GPU using Python 3.7 and the neural network library Keras, with Tensorflow backend and the NVIDIA CUDA Deep Neural Network library (cuDNN).

## Results

The aim of these experiments is to test the learning benefits offered by DDQ and PER - relative to the baseline DRQN architecture that was formulated by Lample et al. and reproduced here - in the context of ViZDoom, an environment where a great deal of state information is hidden from the agent at each time step. Specifically, the authors here studied effects of DDQ and PER on the early phases of learning, using a sizeable frame-skip value. As stated above, a frame-skip value of approximately 10 was used for training of all agents. Given that most research using ViZDoom has studied RL agents using much smaller values of frame-skip (around 5), this work allows a look at RL agent learning rate improvements at higher frame-skip values in the context of this FPS environment. As noted by Braylan et al., using larger frame-skip values greatly accelerates learning \[1\]. Such behavior was also noted by authors here; larger frame-skips led to significant accelerations in learning rate, albeit at the cost of the agent's precision. With larger frame-skip values, the agent can overshoot an enemy when turning toward the target, decreasing the ability of the agent to center on target before shooting. At the other end of the spectrum, too few frame-skips can cause the agent to learn degenerate, repetitive behavior - as noted earlier in this paper - such as assigning higher value, irrespective of input, to the action with the highest variance in value.

For testing, snapshots of each agent - base DRQN, DRQN with DDQ, and DRQN with PER - were taken at 100 game intervals over the course of the full 11500 games of training. These snapshots were then tested in defend-the-center games and, as with training, tasked with gaining as many frags as possible. Specifically, each snapshot was given 100 games to accumulate frags. In addition, three sets of five models each were used in creating snapshot ensembles - one ensemble each for the base DRQN, DRQN with DDQ, and DRQN with PER agent types.

Given the higher frame-skip used, the authors, unsurprisingly, observed that in all cases the performance of agents plateaued after roughly 6000 out of the 11500 games. Given that the subject of interest here is the rate of learning in early development of the agent, this is not an issue. However, it does serve as a basis for further research. In particular, one might ask what the upper limit of performance would be at smaller frame-skip values using the aforementioned DRQN architecture with the addition of either PER or DDQ. This is one of a set of future targets for further investigation.

Concerning metrics of performance, cross entropy loss and K/D ratio are used to measure agents success at various time points in development. Cross entropy loss was used to measure the agent's recognition of an enemy or enemies in the current input frame, calculated using the output of the size-1 dense layer (see Figure 2). K/D ratio is the frags-to-deaths ratio, i.e. the number of total frags by the agent over its total number of lives. Roughly speaking, this translates to frags-to-games-played ratio.

In the case of DRQN with PER, the agent quickly achieved impressive levels of performance, reaching an average K/D of 5.62 after 400 games trained (see Figure 5). The maximum average value (averaged over 100 games for each model and then finding max over all models) for enemy identification cross entropy loss was 0.2196 (see Figure 4 - all loss was normalized by the max loss value observed for each graph), with a mean average K/D of 4.82 and an average K/D standard deviation of 0.614. The snapshot ensemble method improved upon the mean average K/D of DRQN with PER by over a standard deviation, achieving an average K/D of 5.51. Likely, the snapshot ensemble of PER was so successful due to PER's aggressive tendency to fit against challenging samples mixed with the snapshot ensemble method's ability to create similar and yet competitively diverse committees using its cosine annealing learning rate.

|                           | DRQN  | DDQ DRQN | PER DRQN |
|:-------------------------:|:-----:|:--------:|:--------:|
|     Mean Average K/D      | 4.65  |   3.90   |   4.82   |
| Average K/D Standard Dev. | 0.620 |  0.393   |  0.614   |
|     Max Enemy ID Loss     | 0.269 |  0.298   |  0.2196  |

K/D and Enemy ID Loss

For the DRQN with DDQ, the authors report a set of agents characterized by consistency. Cumulatively, these agents had a mean average K/D of 3.90 and an average K/D standard deviation of 0.393, and the maximum average value for enemy identification cross entropy loss was 0.298. The snapshot ensemble method improved upon DRQN with DDQ by over a standard deviation as well, achieving an average K/D of 4.37 .

Using the DRQN architecture mentioned here (without PER or DDQ), the agent achieved a max average K/D of 5.25, never reaching an average K/D to rival the max average K/D observed by the agent with PER. Furthermore, the base DRQN agent had an average K/D standard deviation of nearly double that of the agent with DDQ, illustrating its lack of consistency relative to the DDQ version of itself. To elaborate, the base DRQN agent had a mean average K/D of 4.65, an average K/D standard deviation of 0.620, and the maximum average value for enemy identification cross entropy loss was 0.269. No improvement was observed using the snapshot ensemble of base DRQN's, indicating potentially that increased value overestimation in such an environment led to development of a committee of models that were too dissimilar to allow for advantages offered by ensembling. Note that the DRQN architecture used here is efficient and very competitive, allowing for quick learning of complex tasks. Thus, PER and DDQ should be looked on as boosts to a well tuned learning architecture.

In all experiments, successful enemy detection was achieved by 1000 games by all agent types. In terms of qualitative notes on agent behavior, DDQ DRQN agents were better at developing and maintaining an ammo conservation strategy. Specifically, it was noted that DDQ agents would hold fire longer than other agents, allowing enemies to come closer. This in turn, reduced the possibility of missing a shot. DRQN PER agents, on the other hand, were observed to more quickly respond to enemies attacking from behind. When the agent is damaged, its screen will flash red, and the amount of health remaining is indicated in the lower left of the screen. PER agents were seen to more quickly turn to address and neutralize attackers from behind as well as from the flanks.

# CONCLUSION

In this paper, the authors tested double-Q learning and prioritized experience replay in the context of the ViZDoom reinforcement learning environment. In addition, these methods were coupled with the use of the efficient deep neural network ensemble creation method known as snapshot ensembling. This breaks new ground in the ViZDoom environment, as DDQ and PER had not been tested using this clever, ensembling method - populating members of an ensemble by effectively training a single model, in this case a deep recurrent neural network, with the use of a cosine annealing learning rate. Furthermore, snapshot ensembling yielded significant results, improving DDQ and PER agent performance by over a standard deviation above the mean in both cases.

Following this, there are a number of avenues for further research. One of these consists of examining other DRL models at higher frame-skip values using ViZDoom. To what extent are other DRL model structures (dueling network architectures, actor-critic models, asynchronous variants, etc.) affected? Not only does frame-skipping accelerate training, it can also be looked on as modeling a potential scenario for autonomous agents in physical 3D space. If on a mission for example, a search-and-rescue agent's visual sensors are damaged and only every 11th frame on average of its video feed is a valid image, a reasonable question that arises is: will the agent, trained on a frame-skip distribution with a mean of $x$ be able to cope with the new, shifted frame-skip distribution with mean $x+k$, where $k$ is the additional number of frames, on average, that must be skipped before a valid frame is observed? At what values of $k$ does performance begin to degrade significantly. This is also assuming the distribution shape remains unchanged; what are the limitations of learning from a certain type of frame-skip distribution (uniform, Gaussian, Landau, etc.)? Given its emerging research community and solid RL environment, the authors here note ViZDoom as a promising setting to test further research questions such as these.

# ACKNOWLEDGEMENT

The authors here would like to thank: 1) the ViZDoom development team for their continued maintenance and extension of this remarkable RL framework, and 2) IEEE CIG for their support of the annual ViZDoom Limited and Full Deathmatch Competitions.

[^1]: \*This work was not supported by any organization


abstract: Motivated by the Bagging Partial Least Squares (PLS) and Principal Component Analysis (PCA) algorithms, we propose a Principal Model Analysis (PMA) method in this paper. In the proposed PMA algorithm, the PCA and the PLS are combined. In the method, multiple PLS models are trained on sub-training sets, derived from the original training set based on the random sampling with replacement method. The regression coefficients of all the sub-PLS models are fused in a joint regression coefficient matrix. The final projection direction is then estimated by performing the PCA on the joint regression coefficient matrix. The proposed PMA method is compared with other traditional dimension reduction methods, such as PLS, Bagging PLS, Linear discriminant analysis (LDA) and PLS-LDA. Experimental results on six public datasets show that our proposed method can achieve better classification performance and is usually more stable.
author: Qiwei Xie; Liang Tang; Weifu Li; Vijay John; Yong Hu
bibliography: mybibfile.bib
date: Received: date / Accepted: date
title: Principal Model Analysis Based on Partial Least Squares

# Introduction

For qualitative analysis, high-dimensional datasets provide enough information, but in many cases, not all the measured variables are useful for qualitative model. In addition, traditional statistical methods require the number of variables smaller than the number of samples, otherwise, it will cause the curse of dimensionality . In order to solve these problems, we need to reduce the dimensionality of the dataset before qualitative analysis. Dimension reduction methods such as PCA , LDA and PLS are often used. These methods reduce or eliminate the statistical redundancy and noise between the components of high-dimensional vector data, obtaining a lower-dimensional representation without significant loss of information.

In unsupervised data analysis, PCA is a good tool of dimension reduction, the main idea is to reduce the dimensionality of a dataset in which there are a large number of interrelated variables while retaining as much as possible of the variation present in the dataset . However, PCA can only work in the unsupervised dataset. After adding the sample labels, we need to use supervised methods for analyzing the dataset. LDA is a well-known supervised method for feature extraction and dimension reduction, it achieves maximum discrimination by maximizing the ratio of between-class and within-class distance . An intrinsic limitation of classical LDA is the so-called small sample size problem , different methods have been proposed to solve this problem . One of the most successful approaches is subspace LDA, which applies an intermediate dimension reduction stage before LDA. Among all the subspace LDA methods, the PCA plus LDA (PCA-LDA ) and PLS plus LDA (PLS-LDA ) have received significant attention. Other approaches use the algorithms based on PLS as a dimension reduction.

PLS algorithm has the ability to overcome both the dimensionality and the collinear problems , at the same time, and has exhibited excellent performance for solving the problem of small sample size . However, PLS also has some problems, such as how to obtain more useful information, to enhance the robustness of the model, and to more accurately eliminate redundancy and noise. A solution to these problems is ensemble learning which is derived from the field of machine learning , and can be used for both classification and regression problems. In this study, we are more interested in dimensionality reduction and classification. Compared with the single model, ensemble models, including boosting , bagging and stacked regression , report increased robustness and accuracy and have been successfully applied in the last several years. In order to overcome the over-fitting problem, Zhang et al. used the idea of boosting to combine a set of shrunken PLS models, each with only one PLS component, and called it boosting PLS . On the basis of boosting PLS, some scholars have modified and applied it for spectroscopic quantitative analysis . By using Bagging strategy , many training sets are generated from the original dataset, Bagging PLS trains a model from each of those training sets, the final model can be obtained by averaging the coefficient **B** from each sub-model. From overcoming the disadvantages of MWPLS and iPLS, Xu et al. presented a stack based PLS method using Monte Carlo Cross-validation . Ni et al. have proposed two new stacked PLS which can establish PLS models based on all intervals of a set of spectra to take advantage of the information from the whole spectrum by incorporating parallel models in a way to emphasize intervals highly related to the target property .

After the establishment of the PLS sub-models, various ensemble algorithms for the fusion of the final model are available, mainly including average weighting, cross-validation error weighting and minimum square error weighting rule and so on. In this paper, for adopting Bagging model training method, the dataset is divided into a number of sub-training sets. The PLS models are then employed on these sub-training sets. Subsequently, the coefficients **B** of all the PLS sub-models becoming an asymmetric positive semi-definite matrix **BB**$^{\scriptsize \textmd{T}}$, are fused in a joint matrix. Finally, using the PCA, an eigenvalue decomposition by taking the largest variance model or final projection model is performed. This proposed method is termed as the Principal Model Analysis (PMA). In the subsequent sections, we discussed the relationship between the model parameters (the number of latents, models and remained dimensions) and the classification accuracy. The theory and experiments show that PMA increases the robustness and the generalization ability of the PLS algorithm. Also, PMA can provide a good idea for using the PLS algorithm to semi-supervised dimensionality reduction.

# Background

## Notation

Boldface uppercase and lowercase letters are used to denote matrices and vectors, respectively. Lowercase italic letters denote the scalars. The detailed notations are as follows:

## Overview of PLS and Bagging-based PLS

PLS intends to project the high-dimensional predictor variables into a smaller set of latent variables, which has a maximal covariance to the responses. Given a training set $(\textbf{X}, \textbf{y})$, the decomposition of PLS algorithm is as follows : $$\textbf{X}= \sum_{i=1}^{a}\textbf{t}_{i}\textbf{p}_{i}^{\scriptsize \textmd{T}} + \textbf{E},\  \textbf{y} = \sum_{i=1}^{a}\textbf{u}_{i}\textbf{q}_{i}^{\scriptsize \textmd{T}} + \textbf{F},$$ where $\textbf{t}_{i}$ and $\textbf{u}_{i}$ are score vectors, and $\textbf{p}_{i}$ and $\textbf{q}_{i}$ are loading vectors of **X** and **y**, respectively. **E** and **F** are residuals matrices. The $a$ is the number of feature vectors. The PLS inner relation between the projected score vectors is: $$\textbf{u}_{i} = \bm{\beta}_{i} \textbf{t}_{i}.$$ The detailed algorithm procedures of PLS are as follows:

1.  Initialization: $\textbf{E}_{0}=\textbf{X},\ \textbf{F}_{0}=\textbf{y},\ i=1.$

2.  Computing weight vector: $\textbf{w}_{i}=\textbf{E}_{i-1}^{\scriptsize \textmd{T}}\textbf{F}_{i-1}$, and making $\textbf{w}_{i}$ to be normalization.

3.  Computing the input's score vector: $\textbf{t}_{i}=\textbf{E}_{i-1}\textbf{w}_{i}$ and its loading vector: $\textbf{p}_{i}=\frac{\textbf{E}_{i-1}^{\scriptsize \textmd{T}}\textbf{t}_{i}}{\textbf{t}_{i}^{\scriptsize \textmd{T}}\textbf{t}_{i}}$.

4.  Computing the output's loading vector: $\textbf{q}_{i}=\frac{\textbf{F}_{i-1}^{\scriptsize \textmd{T}}\textbf{t}_{i}}{\textbf{t}_{i}^{\scriptsize \textmd{T}}\textbf{t}_{i}}$, and making $\textbf{q}_{i}$ to be normalization.

5.  Computing the output's score vector: $\textbf{u}_{i}=\textbf{F}_{i-1}\textbf{q}_{i}$.

6.  Computing internal regression coefficient: $\bm{\beta}_{i}=\frac{\textbf{u}_{i}^{\scriptsize \textmd{T}}\textbf{t}_{i}}{\textbf{t}_{i}^{\scriptsize \textmd{T}}\textbf{t}_{i}}$.

7.  Computing residuals matrices: $\textbf{E}_{i}=\textbf{E}_{i-1}-\textbf{t}_{i}\textbf{p}_{i}^{\scriptsize \textmd{T}}$ and $\textbf{F}_{i}=\textbf{F}_{i-1}-\bm{\beta}_{i}\textbf{t}_{i}\textbf{q}_{i}^{\scriptsize \textmd{T}}$.

8.  Updating $i$ to $i+1$, then go back the step 2 until the expected number of latent variables is achieved.

The ensemble learning method aims to improve the accuracy and robustness of traditional algorithms by combining the results of multiple sub-models. Bagging is a simple ensemble learning strategy and is widely used for the classification and regression problems, such as bagging SVM and bagging PLS.

The general PLS method usually shows bad or unstable results on the data with a very large number of collinear x-variables or the data with very limited training samples. By using the bagging strategy, the bagging PLS model could reduce the variance of the original unstable model without increasing the bias. Therefore, bagging PLS usually can achieve much more accurate and stable results than traditional PLS method.

Bagging-based PLS first generates several sub-training sets from the original training set based on the random sampling with replacement method, and then trains a PLS model on each sub-training set separately, finally averages the regression coefficients of all sub-PLS models and uses the averaged regression coefficient for the model prediction. In detail, we suppose that $N$ sub-training sets are generated by random sampling with replacement, and the PLS regression coefficient vector corresponding to each sub-training set is $\bm{\beta}_{i} (i=1,\cdots, N)$. The final regression coefficient of bagging-based PLS can be formulated as: $$\bm{\beta} = \frac{1}{N} \sum_{i=1}^{N} \beta_{i}$$

## Overview of PCA

Principal component analysis (PCA) uses an orthogonal transformation to convert a number of possibly correlated variables into a smaller number of uncorrelated variables called principal components while trying to preserve the data variance. Given a data matrix **X**, computing the covariance matrix **C**, then the projection directions of PCA can be solved by: $$\hat{\textbf{w}} = \arg \max_{\|\textbf{w}\|=1} \textbf{w}^{\scriptsize \textmd{T}}\textbf{C}\textbf{w}.$$ The above problem can be easily solved by the eigendecomposition methods, such as the singular-value decomposition (SVD) algorithm. The detailed algorithmic processes of PCA is as follows :

1.  Data standardization: $$x_{ij}^{*}=\frac{x_{ij}-\bar{x}_{j}}{s_{ij}},\ s_{ij}=\sqrt{\frac{\sum_{j=1}^{m}(x_{ij}-\bar{x}_{j})^{2}}{s_{ij}}},$$ where $\textbf{X}=(x_{ij})_{n\times p}$ is the data matrix with $n$ samples and $p$ variables.

2.  Computing the covariance matrix: $\textbf{C}=\textbf{X}^{\scriptsize \textmd{T}}\textbf{X}$.

3.  Eigendecomposition: $\textbf{C}\textbf{u}=\lambda \textbf{u}$.

4.  Denote the first $m$ eigenvalues as $\lambda_{1} \geq \lambda_{2} \geq \ldots \geq \lambda_{m}$, their corresponding eigenvectors, $\textbf{u}_{1}, \textbf{u}_{2}, \cdots, \textbf{u}_{m}$, are principal components. The number of principal components $m$ can be decided by the cumulative contribution rate of the principal components, i.e., choosing $m$ such that $$\frac{\sum_{i=1}^{m}\lambda_{i}}{\sum_{j=1}^{p}\lambda_{j}} \geq 0.95.$$

# Principal Model Analysis

## Theory and Algorithm

Combing the bagging strategy and PLS, we propose a principal model analysis (PMA) method in this section. The proposed PMA contains two steps. The first step is also to generate $\widehat{N}$ sub-training sets from the original training set with replacement method and the corresponding PLS regression coefficient vector of each sub-training set is denoted by $\bm{\beta}_{i}, i=1,\cdots,\widehat{N}$. Different from the bagging PLS method which just simply averages the PLS sub-models, the second step adopted here is to use the PLS sub-models as the input of PCA algorithm to generate the final PMA model. It is mainly because PCA can effectively find the "major" elements, remove the noise, and reveal the essential structure hidden behind the complex data.

The original PCA algorithm is performed by decomposing the covariance matrix **C**, which is a symmetric and positive semi-definite matrix. However, the whole regression coefficient matrix $\textbf{B}=[\bm{\beta}_{1},\ldots,\bm{\beta}_{n}]$ in the PMA algorithm is not a symmetric and positive semi-definite matrix. So, we need to make the regression coefficient matrix **B** to be a symmetric positive semi-definite matrix. We replace the **B** by **BB**$^{\scriptsize \textmd{T}}$ for the eigenvalue decomposition, and get the most representative models which called principal model as the final PMA model. The optimization of PMA algorithm can be expressed as: $$\label{1_3}
\left\{ \begin{array}{ll}
\textbf{B} = [\bm{\beta}_{1},\ldots,\bm{\beta}_{n}][\bm{\beta}_{1},\ldots,\bm{\beta}_{n}]^{\scriptsize \textmd{T}}\\
\hat{\textbf{w}} =  \arg \max_{\|\textbf{w}\|=1} \textbf{w}^{\scriptsize \textmd{T}}\textbf{B}\textbf{w}
\end{array} \right.$$

The above problem can be easily solved by the singular-value decomposition (SVD) algorithm:

1.  Eigendecomposition: $\textbf{B}\textbf{u}=\lambda \textbf{u}$.

2.  Denote the first $m$ eigenvalues as $\lambda_{1} \geq \lambda_{2} \geq \ldots \geq \lambda_{m}$, their corresponding eigenvectors, $\textbf{u}_{1}, \textbf{u}_{2}, \cdots, \textbf{u}_{m}$, are principal components. The number of principal components $m$ can be decided by the cumulative contribution rate of the principal components, i.e., choosing $m$ such that $$\frac{\sum_{i=1}^{m}\lambda_{i}}{\sum_{j=1}^{p}\lambda_{j}} \geq 0.95.$$

The detailed processes of PMA method are shown as follows.

------------------------------------------------------------------------

**Algorithm PMA** **Input:** Training set and corresponding label vector, the number of PLS latent variables, the number of sub-models, the number of principal models (dim). **Output:** The projection direction of PMA.

------------------------------------------------------------------------

1\. Preprocessing the training set $\mathcal{T}$.. Dividing $\mathcal{T}$ using random sampling with replacement and generating PLS sub-models $\bm{\beta}_{i}$.. Denote $\textbf{B}=[\bm{\beta}_{1},\cdots,\bm{\beta}_{n}][\bm{\beta}_{1},\cdots,\bm{\beta}_{n}]^{\scriptsize \textmd{T}}$, and doing the eigenvalue decomposition in (), sorting the eigenvalues in descending order and rearranging their corresponding eigenvectors.. Denote the rearranged eigenvector matrix as **W**, outputting the final PMA model $B_{PMA}=\textbf{W}(:,1:\textmd{dim})$.

------------------------------------------------------------------------

## Determination of the number of Latent variables

The number of latent variables is an important parameter in the PLS model. There are many approaches to determine the number of latent variables, such as genetic algorithm, F-test and cross-validation methods. Cross-validation methods include K-fold cross-validation (k-CV), leave-one-out cross-validation (LOOCV), Monte Carlo cross-validation (MCCV) and so on. In this paper, we use 10-fold cross-validation method to determine the number of latent variables.

## The sub-models selective rule

For ensemble strategy, usually sub-models who performed better or part of the performance can include more diversity . So Zhou et al. suggested that it may be a better choice for using part of sub-models instead of all of the sub-models . Herein, original training set is arbitrarily divided into tress parts: calibration sets, validation sets and prediction sets . We establish 100 PLS sub-models in the validation sets by sub-sampling and re-weighting the existed calibration samples respectively. The proposed method directly constructs diverse models with virtual samples which are produced by original calibration samples, and this can increase the amount of ensemble diversity when the calibration samples are not enough . Using these 100 sub-models on the validation set, we get 100 different classification accuracies. Then follow the classification accuracy in descending order, take the sub-models with largest classification accuracy participate final ensemble.

## Determination of Dimensions

PCA does an EIG or SVD on a matrix and then generates an eigenvalue matrix. To select the principal components we have to take only the first few eigenvalues. How do we decide on the number of eigenvalues that we should take from the eigenvalue matrix? Usually we adopt accumulative contribution rate automatically retain useful eigenvalues.

Using PMA to reduce dimension is to obtain the scores by projecting the new samples to the direction of the principal models, so the number of the final dimensions is equal to the number of selected principal models. In the experiment, if the parameter of fixed dimensions, which is one of inputs, is greater than or equal to 1, we will use fixed dimensions to select the principal models. Otherwise, if the fixed dimensions greater than 0 and less than 1, we use cumulative contribution rate to obtain the principal models.

From the selection of sub-models can be inferred, the number of final principal models does not need much. Because the classification ability of the selected sub-models are almost the same, so only one principal model almost retains all sub-models classification ability. Therefore, in practical applications, we only take the eigenvector with the largest eigenvalue as the principal model.

# Experimental results

## Data Sets

In order to evaluate the performance of the proposed PMA method, we compare it with the PLS, LDA, PLS-LDA and Bagging PLS methods on three types of data sets:

1.  Four UCI datasets, i.e., Breast data, Spambase data, Gas data, Musk data (Version 1) (obtained from http://archive.ics.uci.edu/ml/datasets.html);

2.  Small data and Imbalanced data;

3.  Raman spectral data (Raman).

The details of these datasets are shown in Table . Before using the datasets, we remove the non-numerical and missing inputs data and convert the class label to a numeric type.

Data Sets

Data Set

Number of Examples
Number of Attributes
Class label
Year

Breast

569
30
1 and 2
1995

Spambase
4601
57
0 and 1
1999

Gas
4782
128
5 and 6
2012

Musk(Version 1)
168
476
0 and 1
1994

small
300
476
0 and 1
1994

imbalanced
7074
476
0 and 1
1994

Raman
925
101
0 and 4
N/A

The data sets "small" and "imbalanced" are randomly sampled from the data set "Musk (Version 1)". The data set "small" is a typical data set with high dimensionality and small samples, where the number of positive and negative samples are the same. The data set "imbalanced" is an imbalanced data set, where the ratio of positive and negative samples is 6:1.

Spectral data set "Raman" is obtained by a standard Raman spectroscope (HR LabRam invers, Jobin-Yvon-Horiba). The excitation wavelength of the used laser (Nd: YAG, frequency doubled) is centered at 532 nm. There are 2545 spectra for 20 different strains available . Herein we select two classes (B. subtilis DSM 10 and M. luteus DSM 348) and use the spectra in the region 1100-1200 in calculations .

## Calculation

Five dimension reduction methods, i.e., PLS, LDA, PLS-LDA, Bagging PLS and PMA, are compared in our experiments. For Bagging PLS algorithm, fifteen models are generated by the random sampling with replacement method and the final model is obtained by averaging these fifteen sub-models. For the PMA method, 100 sub-models are generated from the validation set, and the best fifteen sub-models with higher accuracies are chosen to perform model fusion. Except for the LDA, the number of latent variables in the PLS, PLS-LDA, Bagging PLS and PMA are determined by the 10-fold cross-validation. In the experiment, the dimensionality of the original data is reduced to 1. For fair comparison, the linear Naive Bayes classifier is used to evaluate the results of the above different dimension reduction methods.

For each data set, we randomly choose 49$\%$, 30$\%$ and 21$\%$ samples from the total samples to form the training set, test set, and validation set. The experiments are randomly run 20 times, and the averaged results are recorded.

# Results and Discussion

## Classification performance of different algorithms

This section mainly investigates the classification performance of various algorithms. We report the results on both the training and testing datasets. The classification accuracies accuracies are reported in Table and Table , respectively.

Train Classification Accuracy of Different Comparative Algorithms

Data Set

PLS
LDA
PLS-LDA
Bagging PLS
PMA

Breast

0.9183
0.6750
0.8910
0.9307
0.9632

Spambase
0.8043
0.7763
0.6816
0.8543
0.9070

Gas
0.9704
0.9715
0.7920
0.9703
0.9740

Musk (Version 1)
0.9119
0.7176
0.9039
0.9126
0.9176

small
0.9249
0.9037
0.9299
0.9481
0.9858

imbalanced
0.9726
1.0000
0.9619
0.9769
0.9899

Raman
0.9515
0.7654
0.8508
0.9538
0.9579

The bold value means the maximum accuracy among different methods.

Test Classification Accuracy of Different Comparative Algorithms

Data Set

PLS
LDA
PLS-LDA
Bagging PLS
PMA

Breast

0.9143
0.6428
0.8891
0.9265
0.9545

Spambase
0.8025
0.7607
0.6820
0.8522
0.9034

Gas
0.9694
0.9623
0.7920
0.9694
0.9730

Musk (Version 1)
0.9052
0.7003
0.8980
0.9059
0.9108

small
0.7108
0.6136
0.7128
0.7215
0.7220

imbalanced
0.9003
0.6492
0.8821
0.9091
0.9097

Raman
0.9345
0.6589
0.8400
0.9367
0.9419

The bold value means the maximum accuracy among different methods.

The small-sample-size problem is often encountered in the field of pattern recognition. It may lead to the singularity of the within-class scatter matrix in the LDA. So, for the data sets "small" and "imbalanced", the LDA algorithm shows bad results. PLS shows good overall classification performance. PLS-LDA algorithm firstly removes redundancy and noise in the data set by the PLS method, then performs the LDA algorithm on the PLS dimension reduction features. PLS-LDA shows better results than LDA except for the data sets "Spambase" and "Gas". But PLS-LDA still seems to show over-fitting phenomenon in the data sets "small" and "imbalanced". Because the PLS dimension reduction process may lose some information, the results of PLS-LDA are worse than PLS. The Bagging PLS achieves better results than PLS in the data sets "Breast", "Spambase", "Raman" and "Muskv (Version 1)". Although many sub-models in Bagging PLS provide better performance than PLS, the improvement of Bagging PLS over PLS is not significant because of the average strategy. As observed in the Tables and , all algorithms appear over-fitting phenomenon on the data set "small". Notwithstanding, the proposed PMA algorithm achieves the best results in either training and testing set except for the data set "imbalanced". The superiorities are much more obvious on the data sets "Breast" and "Spambase" .

The above figures are the box diagrams of the accuracy of different algorithms. For the data sets "Breast", "Spambase", "Raman" and "Muskv (Version 1)", the results of LDA algorithm are obviously much worse than other methods. The results of PLS are also unstable on the data set "Spambase". The results of PLS-LDA are worse than others on the data sets "Spambase" and "Gas". In the data set "small", all algorithms show over-fitting phenomenon. Except for the data set "small", PMA algorithm gets more stable results than other methods.

## Investigation on the number of sub-models

From the Figure , we can see that the number of sub-models is less sensitive to the PMA model. In general, the classification accuracies on each data set decreases with the increase of the number of sub-models. It demonstrates that not all of the sub-models are valid. Meanwhile, it is likely to improve the classification performance by choosing some good sub-models. For the data sets "Breast" and "Raman", the number of sub-models greatly affects the classification results. The number of sub-models can be empirically determined by the cross-validation method.

## Impact of PMA dimensions

To investigate the effect of PMA dimensionalities, we show the classification results on different dimensionalities ranging from 1 to 30. As can be seen from the Figure , the classification accuracy on all data sets does not improve with the increase of dimensionality. A possible explanation could be that the first principal component already contains the majority information of the entire data. The results on the data sets "Gas" and "Muskv1" are relatively stable.

## Discussions of the proposed method

The proposed PMA algorithm extends the original Bagging PLS for qualitative analysis. The results on the six data sets show that PMA algorithm can improve the classification accuracy to a certain extent. Model ensemble has many advantages, such as enhancing the robustness. However, the number of sub-models and the number of dimensionalities must be carefully chosen.

# Conclusions

In this paper, we have proposed a PMA method for classification. By means of ensemble strategy, the proposed PMA method fuses the results of PLS sub-models and finds the principal model by performing PCA on the joint coefficient matrix of all sub-models. Experimental results demonstrate that the proposed PMA method can achieve better classification performance than original PLS and Bagging PLS. Our future work will focus on finding more comprehensive evaluation criteria for the selection of sub-models. In addition, we will perform PMA on semi-supervised problems by adding a large number of unsupervised data.