Probabilistic Multi-Layer Perceptrons for Wind Farm Condition Monitoring
Filippo Fiocchi Domna Ladopoulou Petros Dellaportas

empty

[
  Violation of Bell inequalities in an analogue black hole
    Mathieu Isoard
    
========================================================


We provide a condition monitoring system for wind farms,
based on normal behaviour modelling using a probabilistic multi-layer perceptron  with transfer learning via fine-tuning.
The model predicts the output power of the wind turbine under normal behaviour  based on features retrieved from supervisory control and data acquisition (SCADA) systems.  Its advantages are that  (i) it can be trained with SCADA data of at least a few years, (ii) it can incorporate all SCADA data of all wind turbines in a wind farm as features, (iii) it assumes that the output power follows a normal density with heteroscedastic variance and (iv) it can predict the output of one wind turbine  by borrowing strength from the data of all other wind turbines in a farm. Probabilistic guidelines for condition monitoring are given via a CUSUM control chart. We illustrate  the performance of our model in a real SCADA data example which provides evidence that it outperforms other probabilistic  prediction models.
  
  
   
CUSUM control chart,  fine-tuning, heteroskedasticity, normal behaviour modelling, transfer learning. 
  
]




§ INTRODUCTION
 
Due to the environmental and sustainability benefits, many energy producers have shifted their attention towards wind energy; one of the cleanest and fastest-growing sources of renewable energy. However, the severe weather conditions and the often remote locations of wind farms lead to both high operational failure rates and high maintenance costs that account for 20-30% of the total cost related to power generation  <cit.>. There is therefore a  need to reduce the high operational and maintenance costs of wind turbines via a cost-effective and high-precision  condition monitoring system.  Such a system is 
crucial for an early fault detection and thus minimization of the prolonged wind turbine downtime. 

While specific monitoring sensors can be employed to detect incoming faults, wind farm operators are sceptical of employing expensive technology without direct economic justification <cit.>.  Most wind farms have already been equipped with supervisory control and data acquisition (SCADA) systems. These systems record the operational status of individual wind turbines and all their components, such as wind speed, power, rotor speed and blade pitch angles. They produce large amounts of data, which are collected at a high frequency, for example, one Hz, and recorded as a 10-minute averaged interval. Since these systems are already installed, monitoring performance and predicting early faults by extracting features from SCADA data is a widely researched cost-effective measure of failure prevention. 

A typical approach to the problem is to produce a model that predicts the output power of a wind turbine under normal behaviour based on features retrieved from SCADA data; see, for example, <cit.>. Then, one could monitor the wind turbine operation by using the model to inspect whether the observed and model-predicted output powers deviate in some statistical sense. Although this is a sensible, well-known approach, practical and effective condition monitoring of a wind farm requires a modelling perspective with four particular characteristics, outlined below.

First, to accommodate weather fluctuations, the model needs to be able to scale well with the data size since it requires to be trained for at least one calendar year which is 52,560 10-minute intervals.  Second, it should be able to extract all the wealth of information available in the SCADA system  which is expressed in a few dozen features per wind turbine per time interval.
Third, the model needs to provide probabilistic predictions in the form of a predictive density and not only point estimates of output power so that a proper probabilistic assessment of the deviation of the observed from the expected output power can be performed.  The heteroscedastic nature of the predictions is also necessary to properly model the wind turbine output power because low and high wind speeds produce low variance power distributions.
This immediately requires that the predictive density has sufficiently good coverage probabilities in out-of-sample data scenarios.  Last but not least, the model should be able to deal with the peculiarities of wind farm SCADA data recording  in the following sense.  It is common that historical SCADA data may have many missing data in a particular wind turbine because it was out-of-order for a long period of time or it has been recently installed.  A good model should be able to predict the output power of this wind turbine with an inferential procedure that borrows strength from the features and output power of all other wind turbines in the wind farm.

We develop a condition monitoring system based on a model that has all the four necessary characteristics described above based on a probabilistic multi-layer perceptron (PMLP) with transfer learning via fine-tuning.  
We assume that the output power follows a normal distribution with input-dependent mean and input-dependent variance which form our predictive density for each 10-minute interval. 
For the probabilistic condition monitoring we propose the use of CUSUM control charts.
We illustrate  its performance in a real data application by comparing it with two other probabilistic models: a Gaussian process and a Bayesian neural network.  We found that  our model predicts better with respect to root mean square error, mean absolute error and maximum calibration error.  We demonstrate that by transfer learning we can use information from all turbines in a wind farm to improve the prediction of the output power of a single wind turbine. 
We also provide a real data example in which our proposed condition monitoring system expressed via a CUSUM control chart reveals an early warning in a particular wind turbine failure. 

The rest of this paper is organized as follows.  Section <ref> contains related work, Section <ref> presents the proposed methodology, Section <ref> presents the empirical application, Section <ref>  discusses further developments and Section <ref> concludes with our key findings.



§ RELATED WORK
 

Our proposed modelling perspective is based on the notion of normal behaviour modelling, which attempts to produce, conditional on SCADA input features,  predictions of the output power of a wind turbine under healthy conditions and then diagnose possible anomalous performance by comparing the predicted and the observed output powers.  This is an unsupervised learning strategy which differs from another strand of the literature that uses supervised learning and exploits fault instances obtained from operational and events files from a SCADA system.  Thus, in this Section, we will only present related work from articles on modelling normal behaviour.  The more relevant to our work papers that use probabilistic predictions will be presented in subsection <ref>.

Normal behaviour modelling studies vary in their selection of the input features used to identify faults.  Predictions are performed  to the output power <cit.>, the generator's temperature <cit.>, the gearbox bearings' temperature <cit.>, or multiple output variables <cit.>.  


 §.§ Non-probabilistic normal behaviour modelling



A series of articles <cit.>, proposed deep neural networks to model operational characteristics such as the temperature of the gearbox bearings, cooling oil and winding temperature of the wind turbine. 
Autoenconders have been proposed by <cit.> and <cit.> whereas generative adversarial networks combined with autoenconders have been used by <cit.>.  
Other methods include <cit.>, who predicted the generator’s temperature with a nonlinear state estimate technique, and <cit.>, who proposed an adaptive neuro-fuzzy interference system to predict the output power.



 §.§ Probabilistic normal behaviour modelling



Due to their intrinsic probabilistic nature,  Gaussian processes are  common candidates for probabilistic normal behaviour modelling, see for example <cit.>.  The large impediment to their widespread application, namely their computational complexity which increases cubically with the data sample size, is usually being dealt with variational inference with inducing points <cit.>.  However, different input features increase the hyper-parameter dimension since proper implementation requires different kernels in each dimension, which creates both convergence and kernel identification problems.  
Finally, although the issue of heteroscedasticity can be solved by assuming that the error process also follows a Gaussian process <cit.>, it is extremely difficult to resolve all three issues above simultaneously.  Finally, although multi-task Gaussian processes is a modelling perspective that simultaneously exploits the data of all wind turbines, the computational complexity increases quickly and treating missing values from one or more wind turbines is not possible. 
The works of <cit.> have incorporated noise heteroscedasticity whereas the trade-off between heteroscedasticity and computational complexity has been considered by <cit.>.
Recently, <cit.> addressed the necessity of a probabilistic setting for power curve estimation by incorporating prior distributions on the weight parameters of a  long short term memory (LSTM) Bayesian neural network. They improve their model by using a temporal convolutional neural network for temporal feature learning and an embedding layer to map discrete features (e.g., integer years) to dense vectors. The predictions are obtained via Monte Carlo sampling.
Finally, <cit.> considered the problem of predicting output power conditional on future wind speed and direction forecasts and adopted a modelling approach that closely resembles ours.  They employed an improved deep mixture density network by transforming the output energy to the interval (0,1) and obtained probabilistic power predictions by assuming that it follows a mixture of Beta distributions.  As is similarly emphasized in our study, they highlighted the fact that informed decision-making is one of the key benefits of a probabilistic approach to wind power prediction. 



§ PROPOSED METHODOLOGY
 

We present a modelling perspective guided by the need to provide a condition monitoring system that has immediate practical applications in any wind farm.  We first describe a fully connected deep neural network that predicts both the mean and the variance of output power at a particular 10-minute interval.  The computational power of the neural network, together with the stochastic heteroscedastic output power, allows the incorporation of many features obtained from large SCADA datasets that achieve good training based on as many as possible environmental and operational conditions and a statistically sound monitoring system based on both the mean and the variance of the power output.  The final ingredient in our model that is hugely important and necessary in realistic condition monitoring systems is the incorporation of transfer learning which is simultaneously trained in all wind turbines in a wind farm.  Finally, we present a realisation of our proposed probabilistic condition monitoring system based on a CUSUM control chart.  The overall flowchart of the proposed probabilistic condition monitoring
system is presented in Fig. <ref>.


 


 §.§ Probabilistic multi-layer perceptron
 

To model a heteroscedastic output noise we assume that for each  10-minute interval the power output of a wind turbine y conditioned on the SCADA input features 𝐱∈^d_0 follows a normal distribution with unknown mean and variance that both depend on 𝐱, so  
y ∼𝒩(μ(𝐱), σ^2(𝐱)).
We will approximate μ(𝐱) and σ(𝐱)  by a probabilistic multi-layer perceptron (PMLP) with branching prediction heads. We now describe the architecture of our proposed model.

Given an input vector 𝐱= (x_1,x_2,…,x_d_0) that represents the d_0 SCADA input features at a particular time interval, the predicted PMLP power output ŷ∈ follows a normal density with parameters μ(𝐱) and σ^2(𝐱) which are continuous functions of the inputs  𝐱. 
In fact, our proposed PMLP models μ(𝐱) as a linear piecewise function of 𝐱  and σ(𝐱)  as a nonlinear piecewise function of 𝐱.


For a number of k hidden layers  with widths d_1,d_2,…,d_k and input and output dimensions d_0 and d_k+1=1 respectively,
we adopt the linear functions g_i : ^d_i-1→^d_i for i=1,…,k defined as 
 
    g_i(𝐱) = 𝐖_i 𝐱 + 𝐛_i,

 where 𝐖_i ∈^d_i× d_i-1, 𝐱∈^d_i-1 and 𝐛_i ∈^d_i.  
 Furthermore, we adopt the Rectified Linear Unit function ReLU_i : ^d_i→^d_i defined as
 
    ReLU_i(𝐱) = (max{0,x_1},max{0,x_2},…,max{0,x_d_i}),

 and the Softplus function  S : → defined as 
 
    S(x) = log (1+e^x) + δ
 
 for a small δ >0 used for numerical stability needed when x becomes very small.
 These three functions are the basic ingredients of our PMLP, but we note that the functions ReLU and S can be replaced by any other appropriate alternative. 
 
 We now define the branching structure of the PMLP. 
 We use two sets of functions g and ReLU, corresponding to predictions of 
 μ(𝐱) and σ(𝐱)
 and indexed by the superscripts μ and σ respectively.   Different functions g have different sets of parameters  𝐖_i, 𝐛_i where by different functions ReLU is meant that they operate on different dimensions d_i.  We further assume that there exist two corresponding numbers of hidden layers k_μ and k_σ with widths d^μ_1,d^μ_2,…,d^μ_k_μ and  d^σ_1,d^σ_2,…,d^σ_k_σ.  
 Then, the output ŷ of the PMLP  with k_μ and k_σ hidden layers is represented as
 
    μ(𝐱)     = g^μ_k_μ+1∘ ReLU^μ_k_μ∘ g_k_μ^μ∘⋯  g^μ_2 ∘ ReLU^μ_1 ∘ g^μ_1(𝐱) 
    σ(𝐱)     = S ∘ g^σ_k_σ+1∘ ReLU^σ_k_σ∘ g^σ_k_σ∘⋯   g^σ_2 ∘ ReLU_1^σ∘ g^σ_1(𝐱) 
    ŷ   ∼𝒩(μ(𝐱), σ^2(𝐱))

 where ∘ denotes the function decomposition operator.

The branching mode of the PMLP is achieved by setting a layer k^* such that 1 ≤ k^* ≤min{k_μ,k_σ} and setting 
 
    ReLU^μ_i(𝐱)     = ReLU^σ_i(𝐱) 
    
    g^μ_i(𝐱)     = g^σ_i(𝐱) 
    
    d^μ_i    =  d^σ_i

for all 1 ≤ i ≤ k^*.  Thus, before the k^*-th layer there is a common deep neural network; after the k^*-th layer, the network branches into two paths for predicting μ(𝐱) and σ(𝐱) with each path having its own sequence of hidden layers with potentially different widths and depths.
Note that the total sum of hidden layers is k_total= k_μ+k_σ - k^*.
A visual representation that corresponds to the choice of the PMLP architecture given 
k_μ=k_σ = 4, k^*= 3, d_0=34, d^μ_1=d^σ_1 = 100, d^μ_2=d^σ_2 = 80, d^μ_3=d^σ_3 = 40, d^μ_4=d^σ_4 = 20 that was used in the application of Section <ref> is depicted in Fig. <ref>.  We call this particular PMLP architecture A1.

 
 The training of the PMLP is achieved by solving the following empirical risk minimisation problem.  Given n data points (𝐱_i,y_i) ∈^d_0×, i=1,2,…,n, and number and widths of the hidden layers, find ŷ_i that is represented with an PMLP and minimises
 min_𝐖,𝐛ℒ_1
 where 𝐖={𝐖_i }_i=1^k_total, 𝐛={𝐛_i }_i=1^k_total  denoting all parameters, and ℒ_1 is a loss function.  By assuming independence between ŷ_i, the product of normal densities 
 ∏_i=1^n𝒩(μ(𝐱_i), σ^2(𝐱_i))
 can be viewed as a multivariate predictive density so a plausible loss function for our PMLP is the minus logarithmic score, see for example  <cit.>,  defined as 

    ℒ_1 = -∑_i=1^nlog( 𝒩(y_i |μ(𝐱_i), σ^2(𝐱_i)) )

where 𝒩(y | μ,σ)  denotes the p.d.f. of a normal density with mean μ and variance σ^2 evaluated at y.
 



 §.§ Transfer learning via fine-tuning
 
The realistic application of condition monitoring of a wind turbine should necessarily take into account the fact that the wind turbine is part of a wind farm. Thus, instead of training a model for each wind turbine separately, we could use the PMLP model of subsection <ref> to train all wind turbines simultaneously. No particular care is needed other than treating the data so that the corresponding feature values of each wind turbine correspond to the power output of this wind turbine: our proposed PMLP treats the power  output conditional on the corresponding features of each 10-minute period independently of the  power outputs of the other 10-minute periods.  
Therefore, we can apply model (<ref>) to all wind turbines of the wind farm by just denoting 𝐱 as the features from all wind turbines and ŷ as a vector power output for a particular 10-minute period. The loss function (<ref>) is just the sum of log-normal densities of each wind turbine at each 10-minute interval. The mathematical formulation of this  model is as follows.  

Let  𝐱_ij be the input feature vector of the j-th wind turbine  at the i-th 10-minute interval with corresponding  output power y_ij. Then, the  model is defined as 
    μ(𝐱_ij)     = g^μ_k_μ+1∘ ReLU^μ_k_μ∘ g_k_μ^μ⋯  g^μ_2 ∘ ReLU^μ_1 ∘ g^μ_1(𝐱_ij) 
    σ(𝐱_ij)     = S ∘ g^σ_k_σ+1∘ ReLU^σ_k_σ∘ g^σ_k_σ⋯   g^σ_2 ∘ ReLU_1^σ∘ g^σ_1(𝐱_ij) 
    ŷ_ij   ∼𝒩(μ(𝐱_ij), σ^2(𝐱_ij) ).

If we have data for n_j time intervals for wind turbine j for j=1,…,J, the  loss function of model (<ref>) is

    ℒ_2 = - ∑_j=1^J∑_i=1^n_jlog( 𝒩(y_ij|μ(𝐱_ij), σ^2(𝐱_ij)) ).

Notice that the possibly different number of available data n_j  from each wind turbine indicates that this model utilizes all available data from all wind turbines.  This is of huge practical importance in realistic applications since SCADA data typically contain many missing data.

However, by training all data of wind farms in the same model, the amount of available data may become prohibitively vast to allow frequent re-training.  The usual treatment of such huge datasets is to adopt some transfer learning techniques so that knowledge gained from pre-training the model based on all wind turbines data can be used to boost performance in predicting the power of a single wind turbine.   We call such a PMLP model that uses pre-training based on loss function ℒ_2  and then predicts the power of only one wind turbine using fine-tuning and the loss function ℒ_1 
a Large PMLP (LPMLP) model.
When new data arrive, re-training of the PMPL model based on ℒ_2 is not anymore necessary. Moreover, additional important practical advantages arise.  First, consider the very realistic scenario in which 
a wind turbine has fewer data points because, for example, it has been out-of-order for a long period of time or has been recently installed. Normal behaviour modelling of this wind turbine might be very hard or even impossible to achieve.  Our LPMLP model is capable of producing a  predictive density for such low-information wind turbines by using the data from all wind turbines in the wind farm.
Second, transfer learning can be used to improve the predictive power of  model (<ref>) as we empirically show in Section <ref>.

Our proposed transfer learning is achieved via the following fine-tuning.  We first train the LPMLP model (<ref>) with the loss function (<ref>).
Then, we use the parameters of the pre-trained model as initial values, and we  predict the mean and standard deviation of the output power of the wind turbine we are interested in.  Thus, the training uses the loss function (<ref>) that refers to one only wind turbine and is achieved very fast. 
Fig. <ref> depicts the architecture of one  LPMLP used in the application of Section <ref> based on six wind turbines with  k_μ=k_σ = 3, k^*= 3, d_0=34, d_1^σ=d_1^μ=300, d_2^σ=d_2^μ=200, d_3^σ=d_3^μ=100.   
We call this particular architecture A2.  
To show the scalability difference, we note that for our model in Fig. <ref> the input vectors correspond to all features of all wind turbines, and the model has 91,002 parameters whereas the PMPL model of Fig. <ref> has only 16,508 parameters.  

   

For the advantages of transfer learning in similar problems see, for example, <cit.>. Notable theoretical and empirical motivations for introducing transfer learning in the given probabilistic regression context include (i) regularisation:  pre-training the LPML allows learning general features and patterns from a larger amount of data which can be especially beneficial in the case where the operating conditions of the wind farm are extremely diverse.  Thus, it is easier for the model to deal with conditions that have not been observed in the particular wind turbine under inspection; (ii) data sparsity: pre-training on the whole data of a wind farm dataset can help alleviate data sparsity by providing additional data for learning more general but not yet observed representations. For example, see  <cit.> for a discussion of limited data availability for newly built wind farms; and (iii) improved convergence: pre-training the LMPL  facilitates faster converge during fine-tuning on a single wind turbine. 


 §.§ Condition Monitoring

The reason that we propose the PMLP and LPMLP models which have a probabilistic flavour by predicting both the mean and the variance of the output power is to construct a reasonable, probability-based, data-driven condition monitoring system.  This system 
has the usual ingredients that consist of first training the model during a healthy wind turbine operation and then observing both SCADA and output power data.  Anomalous operational behaviour is reported when the observed output power departs, in a probabilistic sense, from the expected predicted normal density under the trained model. Our condition monitoring system is based on testing model adequacy by testing the hypothesis that the observed output power observations have been generated by our trained model.  This requires, of course, some evidence that our model has sufficiently good coverage probability so that in an out-of-sample healthy wind turbine environment the observed power output indeed follow the normal density indicated by the trained model.  As it will be evident in our application study, our proposed PMLP and LPMLP models  do indeed have good coverage probabilities in an out-of-sample large empirical exercise.

Assume that the model training phase has ended and we are now in the live condition monitoring phase. To emphasize that the data is observed sequentially in time,  we will be now using a subscript t rather than i for the time intervals. Assume that we observe T consecutive power outputs y_t, t=1,2,…,T, and we have predictions from our model N(μ(𝐱_t),σ^2(𝐱_t)).  Under the hypothesis of model adequacy, or no-fault, we define  v_t = (y_t - μ(𝐱_t))/σ(𝐱_t) and test whether the sample v_t, t=1,2,…,T comes from a N(0,1) distribution.  For example, we could perform a test on a daily basis for 10-minute intervals where T=144. 

We propose a two-sided tabular CUSUM control chart <cit.> defined as 
    S_H(t)=max{0, v_t-k+S_H(t-1)}
 and 
    S_L(t)=max{0,-k-v_t+S_L(t-1)},
 where S_H(0)=S_L(0)=0.  The reference (or allowance or slack) value of k is traditionally specified with past experience or from simulations based on supervised settings. A common choice is about halfway between the target mean zero and the out-of-control value that we are interested in detecting quickly.  Therefore, if we are interested in detecting a deviance of one standard deviation a plausible choice is k=1/2; see <cit.>.  Recent advances propose an adaptive adjustment of its value <cit.>. Note that  S_H(t) and S_L(t) are viewed as cumulative deviations from zero that are greater than k that are not allowed to take negative values.  If either of them exceeds a decision interval I, the process is considered to be out-of-control.  A reasonable choice of I is five times the process standard deviation <cit.> which is one in our case, so we propose I=5.  For t=1,…,T the test statistics are defined as A_t^H= max{S_H(i), i=1, …, t} and A_t^L=max{S_L(i), i=1, …, T}. The null hypothesis of normal operation in the wind turbine is rejected (with a one-sided test) if A_n=max(A_t: 1 ⩽ t ⩽ n) exceeds its upper critical value, where A_t=max(A_t^H, A_t^L). 



§ EXPERIMENTS
 


 §.§ Data Acquisition and Filtering
 
The application utilises 10-minute SCADA and events data from the six Senvion MM92 wind turbines at Kelmarsh wind farm in the UK <cit.>. The dataset spans from 3^rd January 2016 to 1^st July 2021 and comprises over 1.7 million data points containing 110 variables including date-time, wind speed, bearing temperature, and power output. The data includes 10-minute averages, standard deviations,  minimum and maximum values of all measured variables. The recording of wind speed summary statistics started at 25^th September 2017 so we considered only data entries from this date onwards.
An operational status and events file was utilised for data filtering to ensure consistent modelling of normal behaviour. These files provide valuable insights into the operational conditions of the wind turbines, covering a range of scenarios from technical failures to operational or environmental standbys and warnings. To enable accurate behaviour modelling, it was essential to remove out-of-control condition data records and base our model training solely on filtered data.
The data elimination process is illustrated in Fig. <ref> and <ref> where plots of output powers against wind speeds are depicted for each wind turbine before and after the filtering.  



 

 


All our analyses used as input features 34 operational and environmental variables are listed in Table <ref>. The final filtered dataset across the six wind turbines comprises 966,936 data points, with 189,381 of them belonging to one single wind turbine used in the application Section <ref>. 







 §.§ Application


For all our experiments, we used a randomly selected 80-20 data ratio for the train and test data, respectively. When a validation set was required, the train-validation data ratio was 72-8.
The PMLP and LPMLP models were trained using the Adam optimiser with a learning rate of 0.001, a batch size of 32,  and 100 and 500 epochs respectively. This was done for both A1 and A2 architectures. Additionally, the LPMLP models were fine-tuned for 50 epochs using learning rates of 10^-3 and 10^-4 for A1 and A2 respectively. The Softplus activation function (<ref>) used a δ=0.001. Early stopping on a validation set was employed during all training stages. All results refer to the out-of-sample prediction of the output power of only one randomly chosen wind turbine of the wind farm.

To evaluate the proposed probabilistic PMLP and LPMLP modelling frameworks, we compared their performance to other probabilistic methods. In particular, we used a Gaussian process regression and a Bayesian neural network with stochastic output.  The  Gaussian process inference was based on variational inference with inducing points <cit.> using a radial basis function kernel and 100 pseudo-inputs. This approximate inference procedure was used because the large sample size required low computational complexity methodologies, a problem quite useful in Gaussian process literature; see, for example, <cit.>.
The Bayesian Neural Network was defined as a set of fully connected layers with mean field normal hierarchical priors 
with mean and standard deviation having  𝒩(0, 0.1) and  𝒩(S(0.001), 0.1) 
hyperpriors respectively; here, S is the Softplus function defined in (<ref>) with δ = 0.  It was trained using the Flipout method which re-models the stochastic variational inference implementation strategy as a weight perturbation <cit.> over 100 epochs using early stopping with the loss given by  (<ref>).  

The results shown in Table <ref> are based on 151,504 and 37,877 training and test 10-minutes intervals respectively.  Note that the models have been trained with SCADA data which have the size of nearly three years of wind turbines operation.    We base our comparison with the usual root mean square and mean absolute error defined as RMSE = (n^-1∑_i=1^n (y_i - μ(𝐱_i))^2)^1/2 and 
MAE = n^-1∑_i=1^n |y_i - μ(𝐱_i)| respectively.  Although these metrics provide a good indication on how well our model predicts the output power, it is based only on the predicted means μ(𝐱_i).  An issue of more importance to our condition monitoring methodology is the out-of-sample coverage probabilities that are defined as the probabilities that a confidence interval region will include the true power output.  These are empirically estimated in our test data and  shown in Fig. <ref> where the calibration error, defined as the difference  between observed and theoretical coverage probabilities are plotted for all six models and for twenty different intervals.  The maximum calibrated error, which is the maximum observed deviation, is reported in Table <ref>.

Notice that the architectures A1 and A2 of the PMLP and LPMLP models  are different.  We incrementally built these architectures by inspecting loss functions in training and testing data.  For our empirical exercise, it is of particular interest to investigate whether the scaling of the architecture or the transfer learning is the actual cause of the  model performance improvement.  To illustrate this, Table <ref> and 
Fig. <ref> also contains results of the PMLP model with architecture A2 and of the LPMLP model with architecture A1 without pre-training.  By controlling the network architecture, there is evidence that pre-training improves both RMSE and MAE metrics.















   


We illustrate our proposed unsupervised method for condition monitoring using two different examples, one  in which the examined wind turbine is operating normally and  one in which the process is observed to be out-of-control;  the results of these examples are presented in the CUSUM control charts of Fig. <ref> and <ref> respectively.  

In both cases, observations obtained in daily intervals of T=144 10-minute intervals are being monitored  using CUSUM control charts.  Fig. <ref> presents a typical situation of a good operation of a wind turbine.    

 

In contrast, Fig. <ref> the CUSUM control chart indicates that the process is out-of-control at 07:20AM on the 17^th of January 2018.  By inspecting the operational status and event file of this wind turbine we identified that there is a triggered alarm on the 18^th of January 2018 at 04:22:23AM about a forced outage concerning a fault.  It is, therefore, evident that our monitor system would have identified the fault 21 hours prior to the occurrence.  


 




§ DISCUSSION

 

When the condition monitoring system identifies malfunction, such as the one in Fig. <ref>, the engineers in the field would like to identify which component of the wind turbine is causing the problem. To achieve this, we could extend our method to monitor different variables of the SCADA inputs, such as temperatures or pressures across various parts of the wind turbine. An obvious way to achieve this is to replicate our methodology by replacing the output power with another variable.  A complete condition monitoring system would thus be a collection of CUSUM control charts with different outputs modelled with different LPMLP models.

A further methodological direction would be to replace the fine-tuning of LPMLP which is based on parameter initialisation; see, for example, <cit.>.  We could use the parameter estimates of a pre-trained model to construct informative prior distributions for transfer learning. Then, highly informative posteriors will be available for the model that predicts the power of one single wind turbine.  This is, in essence, a Bayesian neural network with informative priors that replace the vague priors we used in our application.





§ CONCLUSION

We presented a condition monitoring system that can be immediately applied to a wind farm.  We took particular care to accommodate issues that are routinely met in pragmatic wind farm operations. In particular, all SCADA data from all wind turbines collected in  many years can be  used to train our model.  Our monitoring system has the ability to incorporate wind turbines that have many missing data.  It is based on a model that produces heterogeneous predictive densities that are well suited to wind turbine data; its probabilistic nature provides a scientifically sound monitoring system that produces automatic monitoring of a wind farm in the form of a control chart. 

For the data we used, our probabilistic model outperforms other probabilistic methods in terms of RMSE and MAE and has good coverage probabilities in large out-of-sample empirical exercises.  Although it is an unsupervised monitoring system, we were able to illustrate its immediate applicability by inspecting the fault events file after an anomaly is detected.

Future research involves investigating ways to incorporate a further step that will identify which operational characteristic of the wind turbine caused the anomaly detected by the control chart.



§ ACKNOWLEDGMENTS

We would like to thank Dr Evangelos Morfiadakis, CEO at International Wind Engineering, for introducing us to the problem and providing valuable insight; and Dr Charlie Plumley, Senior Performance Engineer at Glennmont Partners, for  helpful discussions. 









