Direct Load Control of Thermostatically Controlled Loads Based on Sparse Observations Using Deep Reinforcement Learning
Frederik Ruelens Bert J. Claessens Peter Vrancx Fred Spiessens Geert Deconinck
cs.LG











































































Shell et al.: Bare Demo of IEEEtran.cls for IEEE Journals





Direct Load Control of Thermostatically Controlled Loads Based on Sparse Observations Using 
  Deep Reinforcement Learning
    Frederik Ruelens, 
        Bert J. Claessens,
		Peter Vrancx, 
		Fred Spiessens,
		and Geert Deconinck 

F. Ruelens and G. Deconinck are with the Department of Electrical
	Engineering, KU Leuven/EnergyVille, 3000 Leuven, Belgium (frederik.ruelens, geert.deconinck@kuleuven.be).
B. J. Claessens is with REstore, 2600 Antwerp, Belgium (bert.claessens@restore.eu).
P. Vrancx is with the AI-lab, Vrije Universiteit Brussel, 1050 Brussels, Belgium (pvrancx@vub.ac.be)
F. Spiessens is with Vito/EnergyVille, 2600 Mol, Belgium.




    December 30, 2023
================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================






This paper considers a demand response agent that must find a near-optimal sequence of decisions based on sparse observations  of its environment.
Extracting a relevant set of features from these observations is a challenging task and may require substantial domain knowledge.
One way to tackle this problem is to store sequences of past observations and actions in the state vector, making it high dimensional, and apply techniques from deep learning.
This paper investigates the capabilities of different deep learning techniques, such as convolutional neural networks and recurrent neural networks, to extract relevant features for finding near-optimal policies for a residential heating system and electric water heater that are hindered by sparse observations.
Our simulation results indicate that in this specific scenario, feeding sequences of time-series to an LSTM network, which is a specific type of recurrent neural network, achieved a higher performance  than stacking these time-series in the input of a convolutional neural network or deep neural network. 




Convolutional networks, deep reinforcement learning, long short-term memory,  residential demand response 























§ INTRODUCTION

Optimal control of Thermostatically Controlled Loads (TCLs), such as heat pumps and water heaters, is expected to play a key role in the
application of residential demand response <cit.>. 
TCLs can use their thermal inertia, e.g. a water buffer or building envelope, as a thermal battery to store energy and shift energy consumption in response to changes in the electricity price or to provide grid services.
Amongst the more important challenges hindering the application of residential demand response is partial observability of the environment <cit.>, where a part of the state remains hidden from the agent due to sensor limitations, resulting in a partially observed control problem.

Model-predictive control (MPC) <cit.> and Reinforcement Learning (RL) <cit.> are two opposing paradigms to solve the optimal control problem of TCLs. 
As such, MPC and RL have developed a set of different techniques to tackle the problem of planning under partial observability.

In MPC, a Kalman filter is often used to  estimate hidden features by exploiting information about the system dynamics and using Bayesian interference.
For example, in <cit.>  Vrettos et al. applied a Kalman filter to estimate the temperature of a building envelope and in <cit.> Kazmi et al. applied a similar approach  to estimate the state of charge of an electric water heater.

RL approaches, on the other hand, store sequences of past interactions with their environment in a memory and extract relevant features based on this memory.
The challenges herein is to consider a priori how many interactions are important to learn a specific task and what exact features should be extracted.
Deep neural networks or multi-layer perceptrons are the quintessential technique for automatic feature extraction in RL <cit.>.
An important breakthrough of automatic feature extraction using deep learning is presented in <cit.>, where  Mnih et al apply a convolutional neural network to automatically extract relevant features based on visual input data to successfully play Atari games.

Finally, by combining RL and MPC, the authors of <cit.> presented a method that trains complex
control policies with supervised learning, using MPC to generate the supervision.
The teacher (MPC) uses a rough representation of its environment and full state, and the learner updates its policy based on the partial state using supervised learning.







§ LITERATURE REVIEW

This section provides a short literature overview of Reinforcement Learning (RL) related to demand response and discusses some relevant applications of deep learning in RL.



 §.§ Reinforcement learning and demand response

An important challenge in tackling residential Demand Response (DR) is that any prior knowledge in the form of a physical model of the environment and  disturbances is not readily available or may be too costly to obtain compared to the financial gains obtained with DR.
As RL techniques can be applied “blind” and consider their environment as a black box, they require no prior knowledge nor do they require a system identification step, making them extremely suited for residential DR.
As a result,  residential DR has become a promising application domain for RL <cit.>.
The most important RL algorithms applied to DR are temporal difference RL, batch RL and more recently deep RL.
The first application of RL to demand response were standard temporal difference methods, such as Q-learning and SARSA <cit.>.
For example, in <cit.>, Wen et al. showed how Q-learning can be applied to a residential demand response setting and
in <cit.>, Kara et al. applied Q-learning to provide short-term ancillary services to the power grid by using a cluster TCLs.
Extending  this work, Mocanu et al. demonstrated how a deep belief network can be integrated in Q-learning and SARSA to extract relevant features <cit.>, allowing for cross-building transfer learning.

In <cit.>, the authors demonstrated how batch RL can be tailored to a residential demand response setting using a set of hand-crafted features, based on domain specific insights.
The authors extended a well-known batch RL algorithm, fitted Q-iteration, to include a forecast of the exogenous variables and demonstrated that it outperformed standard temporal difference methods, resulting in a learning phase of approximately 20-30 days, suggesting that batch RL techniques are more suitable for demand response. 

More recently, inspired by advances in deep learning, the authors extended this approach for a cluster of TCLs using an automatic feature extraction method based on convolutional neural networks <cit.>.
A binning algorithm is used to map the full state of the cluster to a two-dimension representation that can be used as input for the convolution neural network.
Similarly, in <cit.>, François-Lavet et al. applied a convolutional neural network as a function approximator within RL to capture the stochastic behavior of the load and renewable energy production in a microgrid setting with a short-term and long-term storage.





 §.§ Recurrent neural networks and partial observability

In contrast to vanilla neural networks,  Recurrent Neural Networks (RNNs) have an internal state, which is based on the current input state and the previous internal state, allowing the internal state to act as a memory modeling the impact of previous input states on the current task.
This internal state allows the RNN to process sequences of input data, making it a natural framework to mitigate the problem of partial state information.

In practice, however, RNNs have difficulties learning long-term dependencies <cit.>.
An LSTM network is a special type RNN developed by Hochreiter and Schmidhuber in <cit.> that solves the long-term dependency problem, by adding special structures called gates that regulate the flow of information to the memory state.

The application of a RNN within Q-learning, called recurrent-Q, was introduced by Lin and Mitchell in <cit.>, demonstrating  that recurrent-Q was able to learn non-Markovian tasks. 
Extending on this idea, Bram Bakker <cit.> demonstrated  how LSTM  using advantage learning  can solve non-Markovian tasks with long-term temporal dependencies.
In addition to value-based RL, a successful implementation of a policy gradient method with an LSTM architecture to a non-Markovian task  can be found in <cit.>.
Motivated by the promising results of Deepmind with Deep QN <cit.>, the authors of <cit.> demonstrated how an LSTM network can be combined with a deep Q-network for handling partial observability in Atari games, induced by flickering game screens.





 §.§ Contributions

This paper investigates the effectiveness of different deep learning techniques within reinforcement learning for demand response applications that are hindered by sparse observations, making the following contributions. 
We present how  an LSTM network, Convolutional Neural Network (CNN) and multi-layer neural network, can be used within a well-known batch RL algorithm, fitted Q-iteration, to approximate the Q-function, extending the state with historic partial observations.
We demonstrate their performance for two popular embodiments of flexible loads, namely a heat pump for space heating and an electric water heater.

The paper is structured as follows. Section <ref> states the problem and formalizes it as a  Markov decision process. 
Section <ref> explains how these deep learning techniques can be used to extract relevant features based on sequences of observations and used within a batch RL. Section <ref> describes the different deep learning architectures.
Section<ref> presents the simulation results of two flexibility carriers and finally Section<ref> draws conclusions and discusses further work.



§ MARKOV DECISION-MAKING FORMALISM


This section states the problem and presents the formalism to tackle it.




 §.§ Problem statement

In most complex real-world problems, such as demand response, an agent cannot measure the exact full state of its environment, but only a partial observation of the state.
Depending on how good this partial observation can be used to model future interactions, using partial state information may result in sub-optimal policies.
This paper presents two demand response applications that are hindered by partial observability, where the agent cannot measure the state directly, but has to extract relevant features based on how much energy the application consumed and how much it lost.
In our  first experiment, we consider a heat-pump agent that can only measure its electricity consumption and outside temperature. 
In the second experiment, we consider an electric water heater agent with partial state state information, consisting of its measured electricity consumption and the flow rate and temperature of the tap water exiting the water buffer.


To tackle this challenge, we will first formalize the underlying problem as a Markov decision process and then introduce the concepts of partial state information.




 §.§ Formalism


At each discrete time step k, the full state of the environment evolves as follows: x_k+1 = f(x_k,u_k,w_k) with w_k  a realization of a random disturbance drawn from a conditional probability  distribution  and
u_k∈ U the control action.
Associated with each action of the agent, a cost c_k is provided by c_k=ρ(x_k,u_k,w_k), where ρ is a cost function that is a priori given.

The goal of the agent is to find an optimal  control policy h^*:X→ U that minimizes the expected T-stage return for any state in the state space.
Value-based RL techniques characterize the  policy h is  by using  a state-action value function or Q-function:

    Q^h(x,u) = w∼[ρ(x,u,w) + J^h_T(f(x,u,w)) ],


The Q-function is the cumulative return starting from  state x, taking action u, and following h thereafter.
Given the Q-function, an action for each state can be found as:

    h(x) = arg min  Q^h(x,u).

This paper applies a value-based batch RL technique to approximate the Q-function corresponding to the optimal policy  based on an imperfect observation of the true state.




 §.§ Partial state

It is assumed that the state space X  measured by the agent consists of three components: timing-related state information X^, controllable state information X^, and exogenous (uncontrollable) state information X^. 
In this work the timing related is given by the current quarter in the day ∈ X^ = {1,…,96}, which allows the agent to capture time-varying dynamics.
The controllable state information ∈ X^ comprises the operational 
measurements that are influenced by the control action.
In reality, mosts agent can only measure a partial observation  of the true state _k, resulting in a partially observable Markov decision problem.
The exogenous information  is invariant for control actions u_k, but has an impact on the dynamics. Examples of exogenous variables are weather conditions and demand profiles (e.g heat demand).

Thus, the state measured by the agent at step k is given by:

    _k = (,,).

Note that since (<ref>)  only includes part of the true state, it becomes impossible to model future state transitions, making the state non-Markovian.



 §.§ Action


At each time step, a  demand response agent can request an action u_t∈ [0,1]: either to switch OFF or ON.
To guarantee the comfort and safety constraints of the end users, each TCL is equipped with an overrule mechanism (or thermostat).
The backup function  B:X × U ⟶ maps the requested control action u_k∈ U taken in  state x_k to a physical control action ∈: 

    = B(x_k,u_k).
 
The settings of the backup function B are unknown to the learning agent, but the resulting action    can be measured by the learning agent.



 §.§ Cost

This papers considers a dynamic pricing scenario where an external price profile is known deterministically at the start of the  optimization horizon:

    c_k= ρ(,λ_k) = λ_k Δ t,

where λ_k is the electricity price at time step k and Δ t is the length of a control period.




§ BATCH REINFORCEMENT LEARNING


Given full observability, batch RL algorithms start with a batch of four tuples of the form:  (_k,u_k,_k', ),  where _k represents the true state of the problem.

According to the theory of partial observable Markov decision processes <cit.>, the optimal value function at time step k depends on the partial state observations of all proceeding periods.
However, since these observations accumulate over time, it is important to capture sufficient statistics, thats  a history length h  which summarizes the essential content of the measurements.
As such, this paper tackles the problem of partial observability by augmenting the state vector with a sequence of partial state observation, requested actions and physical actions of the last h observations:

    _k = (, _k,)

with _k give by:

    [,…, _k-h],[_k-1, …, _k-1-h],[u_k-1,…, u_k-1-h].

As a results, this paper starts from a bath of four tuples given by:  {(_k, u_k, _k, )}_k=1^ℳ, where _k represents the augmented state.
An important challenge is to learn how to extract relevant features from  in a scalable way.





 §.§ Fitted Q-iteration

This paper applies  fitted Q-iteration <cit.> to obtain an approximation of the Q-function Q^*(,u).
Fitted Q-iteration iteratively approximates the Q-functions for each state-action pair using its corresponding cost and  the approximation of the Q-function from the previous iterations.
To leverage the availability of forecasts of exogenous information , e.g. outside temperatures, we use the extension of fitted Q-iteration as presented in <cit.>, which replaces the observed exogenous
information by its forecasted value   (line <ref> in Algorithm <ref>).




















In order for  Algorithm <ref> to work, we need to select an approximator architecture  (step <ref>) that is able to learn relevant features from sequences of input data and  that can generalize the Q-function.

  










§ DEEP LEARNING APPROXIMATORS


This paper investigates the effectiveness of the following deep learning approximators when combined with fitted Q-iteration.



 §.§ Deep neural network 

It has been shown that a neural network with a single layer is sufficient to represent any function, but the layer may become infeasible large and  may fail  to train and generalize correctly.
To overcome these two challenges, deeper networks are used as these networks can reduce the number of units to represent the function and can reduce the generalization error.   
Fig.  <ref>(a) illustrates the neural network as used in this paper, consisting of an input layer given by (_k,u_k), two hidden layers with rectified linear units (ReLUs), and one linear output layer, representing the approximated Q-function. 



 §.§ Convolutional neural network

CNNs have been successfully applied to extract features from image data, represented as a 2D grid of pixels.
In this paper, we consider  a time series and convolve a 1D filter of length N over the time-series in the state (<ref>).
A sketch of the applied CNN can be seen in Fig. <ref>(b), which consists of two components that are merged to output a singe value. The first component is a dense neural network which takes the timing-related information,  exogenous information and action as input.
The second components is a CNN which takes the time-series as input (<ref>).
For each sequence, the network consists of two layers containing eight 1D filters of length 4 followed by a ReLU, which is downsampled by using an average pooling layer.



 §.§ Long short-term memory



  §.§.§ Background

An LSTM network (Fig. <ref>) consists of LSTM nodes that are recurrently connected to each other.
Each LSTM node has an internal recurrence or memory cell C^(t) and a system of gating units that controls the flow of informations.

 For each step t of the sequence ^(1),…,^(t),…,^(h), the  resulting action of the forget gate f^(t), input gate i^(t) and output gate o^(t) of a single LSTM nodes is provided by:
 
    f^(t) = σ(W_f[h^(t-1),^(t)] + b_f),
       i^(t) = σ(W_i[h^(t-1),^(t)] + b_i),
       o^(t) = σ(W_o[h^(t-1),^(t)] + b_o),

 where W_f,W_i,W_o and b_f,b_i,b_o are the weights and biases of the forget, input and output gate, σ denotes the logistic sigmoid function and ^(t) denotes the current element of sequence (<ref>), with the time step index t ranging from 1 to h.
 
 
 The  internal memory cell of the LSTM node is updated as:
 
    C^(t) = f^(t)*C^(t-1) + i^(t) *C^(t),

 where C^(t) and C^(t-1) are the current and previous memory  state and * denotes a pointwise multiplication operator.
 Note that the new memory C^(t) is defined by the information it forgets from the old state f^(t)* C^(t-1) and remembers from the current i^(t)* C^(t).
    
In a last step, a hyperbolic tangent function is applied to the memory cell and multiplied with the output o^(t), which defines what information to output.
  
    h^(t) = o^(t)* tanh(C^(t)),


This gating mechanism allows the LSTM network to store information about the state for long periods of time and protects the gradient in the cell from harmful changes during training related to the vanishing or exploding gradient problem of RNN <cit.>.











  §.§.§ Approximator architecture

The approximator architecture consists of two components: an LSTM network and a standard multi-layer perceptron (Fig. <ref>).
The first part of the input, corresponding to the LSTM component, contains the historic information of the partial state .
For each k=h+1,…,#ℳ, the input of the LSTM network is given by the following sequence:

    [    _k;  _k-1; u_k-1 ]_^(1)_k
    ,…,
    [ _k-h+1;   _k-h;  u_k-h ]_^(h)_k

The history depth h defines how much time steps the network can see in the past to compute its approximation of the Q-function.
The length of the memory cell d_cell represents an important hyper parameter and defines how many knowledge can be encoded. 
 As can be see in Fig. <ref> only the content of the last memory cell h_t is used as an input for the next layer.
 
The second part contains the time-related information, exogenous information and action: , , u .
The outputs of both components are combined to form a singe architecture, which is followed by two fully connect layers with Rectified Linear Unit (ReLU) activation functions.
A final linear output layer approximates the final Q-function for the provided state-action pair.  








§ SIMULATION EXPERIMENTS


This section evaluates the performance of combining the presented deep learning techniques with Alogirthm 1 for two  providers of demand flexibility  exposed to a dynamic energy price.




 §.§ Simulation framework

At the end of each simulation day, Algorithm <ref> is used to compute  a new policy based on current batch and electricity price for the following day. 
The RL  agent starts with an empty batch and alternates exploration and exploitation according to a decreasing exploration probability: ε_d = 0.75^d, where d denotes the current episode.

All experiments are repeat 10 times starting form a different random seed, resulting in different exploration probabilities and stochastic disturbances. The following results indicate the average of these simulation experiments, where a confidence bound (±2σ) is indicated by a shaded area, representing a 0.95  probability that the solution lies in the shaded area.


The average simulation time for one day (Algorithm <ref>) is about 1.5 hour[Simulation hardware: Xeon E5-2680 v2 processor with 15 GiB memory (Amazon elastic cloud instance type: c3.2xlarge).] using Keras with Theano as backend.



 §.§ Experiment 1: Space heating

Similar as in <cit.>, a second-order heat-pump model (C_a=2.441MJ/^∘C, U_a=125W/^∘C, C_m = 9MJ/^∘C, H_m=6.863kW/^∘C) with outside temperatures from <cit.> is used to simulate the temperature dynamics of a residential building with a heat pump. The heat pump has a maximum electric heating power of 2.3kW and the minimum and maximum comfort settings are set to 20^∘C and 23^∘C.
To model stochastic impact of user-behavior we  sample an exogenous temperature disturbance from 𝒩(0,0.025).
The time resolution of the dynamics is 60 seconds and of the control policy is 15 minutes. 

The state vector describing the environment is defined as:

    _k = (,, , , ),

where _k contains timing information,  the air temperature,  the virtual mass temperature,  the outside temperature and  an exogenous disturbance.
As stated in the problem description, it is assumed that the RL agent cannot measure the air and mass temperature of the building, resulting in a partial observed control problem.
As such, we construct the following augmented state vector:

    _k = (_k,    [_k-1,…, _k-h],
       [u_k-1,…, u_k-h], 
       [_k-1,…, _k-h], _k ),

which includes three time series of lenght h=20. 










  §.§.§ NN Architecture

The neural network consists of three dense layers with 50 neurons with ReLU activation functions, followed be a linear output unit. The neural network was trained using RMSprop with a minibatch size of 32.



  §.§.§ CNN Architecture

The network consists of two components, namely a CNN and dense network.
The CNN  component consists of two 1D convolutions (along the time dimension) that are each followed by an average pooling layer.
The dimension of the first filter is (L×3), where L  is the filter length and 3 is number of input sequences and the dimension of the second filter is L×1. Both filters have a filter length of 4.
The dense network processes the time-related information, exogenous information and action.
Both components are merged and followed with two layers with 20 neurons and a single output layer. 
All layers use ReLU activation function except for the output layer that uses a linear function.



  §.§.§ LSTM Architecture

The input to the LSTM network is provided by the sequence: 

    [  _k-1,; u_k-1,;   _k-1 ]
     ,…, 
      [  _k-h,; u_k-h,;   _k-h ]

 and the NN is provided by _k, _k and u_k.

For the heatpump experiment the best results were obtained with the history depth h set to 20 time steps (quarters) and the length of each LSTM memory cell d_cell set to 8. 



  §.§.§ Convergence

Fig. <ref> depicts the cumulative cost using function approximators (top) and daily average outside temperature (bottom). The no control strategy activates the backup controller, without setting a control action, and can be seen as a worst case scenario as it is agnostic about the electricity price.
An upper bound is computed by considering the full state information as defined in (<ref>).
In addition to LSTM with partial state information, the figure depicts the  cumulative of using an ensemble of extremely randomized trees (or ExtraTrees) <cit.>.
The number of trees in the ensemble was set to 100 and the minimum sample size for splitting a node to 5.
Our results indicate that the ExtraTrees approximator was not able to extract relevant features from the partial state information and performed only 1.5% better than the  no control strategy.
In contrast, the LSTM  approximator was able to extract relevant features and achieved a reduction of 5.5%.
 
Fig. <ref> shows the daily cost (top) obtained with Algorithm <ref>, using a partial state information using a neural network, CNN and LSTM network as a function approximator.
The middle graph indicates the scaled cost which is c calculated as follows: (c - c_f)/(c_nc - c_f), where c_f is the result of using the full state information and c_nc of using the no control strategy, resulting in c=0 for the full state strategy and c=1 for the no control strategy.
This figure indicates Algorithm 1 obtained a scaled cost of  0.37 using LSTM, 0.66 using NN and 0.82 using CNN.
The bottom graph compares the resulting control policies of LSTM, CNN en NN with the control policy of the full state using a euclidean distance. 
Although NN achieved a better performance than CNN, the resulting policy of CNN and LSTM are closer to the policy of the full state.
We speculate that the CNN and LSTM learned a better representation of the full state than the NN, since the NN achieved a low cost by lowering the air temperature to minimum temperature without reacting to the price.









  §.§.§ Daily results

A more qualitative interpretation of our results can be seen in Fig. <ref>.
The figure shows the power consumption and the corresponding  daily price profiles. It can be seen that the learning agent successfully postponed its power consumption to low price moments, while satisfying the comfort constraints. 













 §.§ Experiment 2: Electric water heating

The second experiment considers an electric water heater with a water buffer of 200 liters and a daily average water consumption of 100 liter. The minimum and maximum water temperature is set to 45^∘C and 65^∘C. The water heater is equipped with a thermostatic mixing value to assure a constant requested temperature of 45^∘C. The water heater has an electric power rating of 2.3kW and a built-in backup controller as defined in <cit.>.
The time resolution for the dynamics is  5 seconds and the time resolution for the control policy is 15 minutes. 

The full state vector of the electric water heather is defined by:

    _k = (_k,T_k^1,…,T_k^|ℒ|, d_k),

where T_,t^i is the temperature corresponding to the ith layer and d_k is the current tap demand.
During our simulation, a non-linear stratified model with 50 layers is used to simulate the temperature gradient along the water tank and stochastic tap water profiles are used based on <cit.>.

In a previous paper <cit.>, the authors considered that the agent could measure a imperfect state through eight temperature sensors.
In this experiment, however, it is assumed that the buffer is not equipped with a set of sensors to measure the different temperatures inside the water buffer. 

As a result, we define the following augmented state vector:

    _k = (_k,   [u_k-1,… u_k-h],[_k-1,…_k-h]
       [_k,…_k-h+1] , [T_k^,…, T_k-h+1^] ),

where _k contains timing information, u_k is the requested control action,  the actual action, and ṁ_k and T_k^ are the mass flow rate and temperature of the water exiting the water buffer.
Note that [_k-1,…_k-h] represents the electricity consumption of the boiler and [_k,…_k-h+1] , [T_k^,…, T_k-h+1^] represents the energy flowing out of the boiler. 











  §.§.§ (C)NN Architecture

The NN and CNN architecture are identical as in the previous experiment with the exception that the filters size of the first convolutional layer is 4×4, because now we have 4 input sequences.



  §.§.§ LSTM Architecture

The input to the LSTM network is provided by the sequence:

    [  _k-1,; u_k-1,;   ṁ_k,;   T_k^ ]
    ,…, 
    [    _k-h,;    u_k-h; ṁ_k-h+1,; T_k-h+1^ ]


For the boiler experiment the best results were obtained with 
the history depth h set to 40 time steps (quarters) and the length of each LSTM memory cell d_cell set to 12. 






  §.§.§ Daily results

For the electric water heater scenario, we only offer qualitative results (Fig. <ref>).
It shows the daily power consumption of an  electric water heater and corresponding price profiles. It can be seen that the learning agent required four weeks of learning before obtaining reasonable policies (lower row of graphs).
A final comparison between using a CNN or LSTM network as a function approximator can be seen in Fig. <ref>, indicating that using a CNN resulted in a cost reduction of 5.5% and using a LSTM network in 10.2%. The results of FQI-NN were omitted because we were to able to stabilize the learning of the NN.



§ CONCLUSIONS AND FUTURE WORK


In this paper, we demonstrated the effectiveness of combining different deep learning techniques with reinforcement learning for two demand response applications that are hindered by sparse observations of the true state.
Since these sparse observation result in a non-Markovian control problem, we extended the state with sequences of past observations of the state and action. 
	
In a first experiment, we considered an agent that controls a residential heating system under a dynamic pricing scenario, where the agent can only measure its electricity consumption, control action and outside temperature. 
Our simulations indicated that reinforcement learning with long short-term memory (LSTM) performed better than other techniques such as a  neural network, convolutional neural network and ensemble of regression trees, when sparse observations are used.
In our second experiment, we considered an agent that controls a residential electric water heater with a hot storage vessel of 200 liter.
In this scenario, the agent can only measure its electricity consumption, control action and flow and temperature of the tap water exiting the storage vessel.
The simulation results indicated that the LSTM network outperformed the  convolutional network and deep neural network. 

We speculate that the higher performance of the LSTM network comes from its internal memory cell which can act as an integrator.
This internal memory cell allows the LSTM network to process sequences of sparse observations and extract relevant features from it that can represent the underlying state of charge (or energy level) of the application.
A potential direction of future research would be to extract and visualize the relevant features that are being detected by the LSTM network and CNN that would lead to a better performance and understanding.





































































IEEEtran  
 


