Multi-Agent Q-Learning for Minimizing Demand-Supply Power Deficit in Microgrids
Raghuram Bharadwaj Diddigi D. Sai Koti Reddy Shalabh Bhatnagar










Multi-Agent Q-Learning for Minimizing Demand-Supply Power Deficit in Microgrids
    Raghuram Bharadwaj D.
	IISc, Bangalore, India

	 raghu.bharadwaj@csa.iisc.ernet.in
	D. Sai Koti Reddy  
	IISc, Bangalore, India 

		 danda.reddy@csa.iisc.ernet.in
	Shalabh Bhatnagar
	IISc, Bangalore, India

		shalabh@csa.iisc.ernet.in
    December 30, 2023
==============================================================================================================================================================================================================================================





We consider the problem of minimizing the difference in the demand and the supply of power using microgrids. We setup multiple microgrids, that provide electricity to a village. They have access to the batteries that can store renewable power and also the electrical lines from the main grid. During each time period, these microgrids need to take decision on the amount of renewable power to be used from the batteries as well as the amount of power needed from the main grid. We formulate this problem in the framework of Markov Decision Process (MDP), similar to the one discussed in <cit.>. The power allotment to the village from the main grid is fixed and bounded, whereas the renewable energy generation is uncertain in nature. Therefore we adapt a distributed version of the popular Reinforcement learning technique, Multi-Agent Q-Learning to the problem. Finally, we also consider a variant of this problem where the cost of power production at the main site is taken into consideration. In this scenario the microgrids need to minimize the demand-supply deficit, while maintaining the desired average cost of the power production. 










§ INTRODUCTION


Electricity is one of the most important components of the modern life. According to a recent survey, there are a total of 18,452 un-electrified villages in India <cit.>. Providing electricity to these villages is difficult for a number of reasons. The village may be situated very far away from the main grid and it would be difficult to establish a direct electrical line between the main grid and the village. Also, due to increasing global warming, we want to make less use of fossil fuels for the generation of power. Our objective in this work is to provide a solution to electrifying these villages.

The concept of smart grid <cit.>, is aimed at improving traditional power grid operations.  It is a distributed energy network composed of intelligent nodes (or agents) that can either operate autonomously or communicate and share energy. The  power grid is facing wide variety of challenges due to incorporation of renewable and sustainable energy power generation sources. The aim of smart grid is to effectively deliver energy to consumers and maintain grid stability.

A microgrid is a distributed networked group consisting of renewable energy generation sources with the aim of providing energy to small areas. This scenario is being envisaged as an important alternative to the conventional scheme with large power stations transmitting energy over long distances. The microgrid technology is useful particularly in the Indian context where extending power supply from the main grids to remote villages is a challenge. While the main power stations are highly connected, microgrids with local power generation, storage and conversion capabilities, act locally or share power with a few neighboring microgrid nodes <cit.>.
Integrating microgrids into smartgrid poses several technical challenges. These challenges need to be addressed in order to maintain the reliable and stable operation of electric grid.

Research on smartgrids can be classified into two areas -  Demand-side management and Supply-side management. Demand side management (DSM) (<cit.>) deals with techniques developed to efficiently use the power by bringing the customers into the play. The main idea is to reduce the consumption of power during peak time and shifting it during the other times. This is done by dynamically changing the price of power and sharing this information with the customers. Key techniques to address DSM problem in smart grid are peak clipping, valley filling,  load shifting <cit.>. In <cit.>, Reinforcement Learning (RL) is used in smart grids for pricing mechanism so as to improve the profits of broker agents who procure energy from power generation sources and sell it to consumers.  

Supply-side management deals with developing techniques to efficiently make use of renewable and non-renewable energy at the supply side. In this paper, we consider one such problem of minimizing Demand-Supply deficit in microgrids. In <cit.>, authors developed an MDP and applied Dynamic optimization methods to this problem. But when the model information (the renewable energy generation in this case) is not known, we cannot apply these techniques.

In our current work, we setup microgrids closer to the village. These microgrids has power connections from the main grid and also with the batteries that can store renewable energy. Owing to their cost, these batteries will have limited storage capacities. Each microgrid needs to take decision on amount of renewable energy that needs to be used at every time slot and the amount of power that needs to be drawn from the main grid. Consider a scenario where, microgrids will use the renewable energy as it is generated. That is, they do not store the energy. Then, during the peak demand, if the amount of renewable energy generated is low and power obtained from the main grid is also low, it leads to huge blackout. Thus, it is important to intelligently store and use the renewable energy. In this work, we apply Multi-agent Q-learning algorithm to solve this problem. 



§ ORGANIZATION OF THE PAPER
	
The rest of the paper is organized as follows. The next section describes the important problems associated with the microgrids and solution techniques to solve them. Section III presents the results of experiments of our algorithms. Section IV provides the concluding remarks and Section V discusses the future research directions.



§ OUR WORK

In this section, we discuss two important problems associated with the microgrids. We first formulate the problem in the framework of Markov Decision Process (MDP), similar to the one described in <cit.>. We then apply cooperative Multi-Agent Q-Learning algorithm to solve the problems.



 §.§ Problem 1 - Minimizing Demand-Supply Deficit


Consider a village that has not been electrified yet. We setup microgrids close to the village, and provides electrical connections to the village. It also has power connections from the main grid. Decision on power supply is generally taken in time slots, for example every two hours. Microgrids do not have non-renewable power generation capabilities. They have access to the renewable power and needs to obtain the excess power from the main grid. For ease of explanation, we consider two renewable sources - solar and wind. At the beginning of each time slot, microgrids obtain the power demand from customers with the help of the smart meters <cit.>. These microgrids need to take decisions on amount of renewable power to be used so that the expected long term demand-supply deficit is minimized.

One natural solution without the use of storage batteries is to fully use the solar and wind power generated in each time slot. The excess demand will be then requested from the main grid. If the requested power is less than or equal to the maximum allotted power, the main grid transfers this to the microgrids. Other wise, it transfers the maximum allotted power. This idea is described in Algorithm 1.





Our objective in the current work is to do better than the above described solution. Microgrids are equipped with batteries that can store the renewable power. That is, we deploy the microgrids at the sites where there is availability of renewable resources. The goal is to intelligently use the stored power in batteries and take the optimal action at every time slot to minimize the demand-supply deficit.

We formulate the problem in the framework of Markov Decision Process (MDP). MDP <cit.> is the most popular mathematical framework for modeling optimal sequential decision making problems under uncertainty. A Markov decision process is defined via tuple < S; A; P;R >, where S is the set of states. A is the set of actions. P respectively the probability transition matrix. P_a(s,s^') is the probability that by taking action a in the state s, the system moves into the state s^'. R is the reward function where R_a(s,s^') is the reward obtained by taking action a in the state s and the s^' being the next state. In infinite horizon discounted setting <cit.>, the goal is to obtain a stationary policy π, which is a mapping from state space to action space that maximizes the following objective:


    ∑_t = 0^∞γ^tR_a_t(s_t,s_t+1),

where a_t = π(s_t), and γ∈ [0,1] is the discount factor.

We model Power Demand as a Markov process. That is, the current Demand depends only on the previous value. The Demand value, and the amount of power left in the batteries form the state of the MDP. That is, 

    state = [Demand, solar_batterylevel, wind_batterylevel].


Based on the current state, the microgrids need to take decision on the number of units of power to be used from their batteries, and the number of units of power that is needed from the main grid. Note that the power from the main grid is fixed and bounded during all the time periods, while power from the  renewable sources is uncertain in nature. 


    Action = [solar_power, wind_power, main_power],


The state evolves as follows:

    solar_batterylevel    = solar_batterylevel - solar_power 
       + solar(),

and

    wind_batterylevel    = wind_batterylevel - wind_power 
    
         + wind(),

where
solar() and wind() are the solar and wind power generation policies. 
Our objective is to minimize the expected long term discounted difference in the demand and supply of the power. So, the single stage Reward function is as follows:


    R = -   (Demand - 
       (solar_power+wind_power+main_power))^2

When there is uncertainty in the system, like the renewal energy generation in this case, traditional solution techniques like value iteration and policy iteration cannot be applied. Reinforcement Learning provides us with the algorithms which can be applied when the model information is not completely known. One such popular model-free algorithm is Q-learning <cit.>. However, microgrids applying Q-Learning independently is not a feasible solution, as they are working towards a common goal. Hence we apply co-operative Multi-Agent Q-Learning algorithm <cit.> to solve the problem. At every time slot, the microgrids exchange the battery level information among themselves. Then they apply Q-Learning on the joint state and obtain the joint action. Each microgrid then selects its respective action. Let the joint state at time k be s_k. We select a joint action, based on an ϵ-greedy policy. An ϵ-greedy policy is one where we select the action that gives maximum reward with probability 1-ϵ and a random action with probability ϵ. Let it be a_k and so we move to a new state s_k+1. Let the single stage reward obtained be r_k Then the Q-values are updated as follows:

    Q(s_k,a_k) = Q(s_k,a_k) + 
    α[r_k+γ max_aQ(s_k+1,a) - Q(s_k,a_k)],

where α∈ [0,1] is the learning parameter and γ∈ [0,1] is the discount factor.

We update this Q-values until convergence. At the end of this process, we will have an optimal policy that gives for each state, the optimal action to be taken. The idea is described in  Algorithm 2. 






 §.§ Problem 2 - Balancing Demand-Supply Deficit and cost of Power Production

In the above formulation, cost of the non-renewable power production at the main grid site is not taken into consideration. Therefore, it is natural to use the maximum alloted power from the main grid and the optimization is done on the amount of power drawn from the solar and the wind batteries. In this formulation, we put a cost on amount of power that can be obtained from the main grid. That is, we modify our single stage Reward as follows:

    -c*(Demand - (solar_power + wind_power+ 
    
     main_power))^2+(1-c)*(main_power)^2.
    
     c ∈[0,1],



where c is the parameter that controls the demand-supply objective and the main grid production cost. Note here that when c =1, it is same as the problem 1. In this formulation, our objective is not only to minimize the demand-supply deficit, but also maintain a desired average production of power at the main site. Similar to the above problem, we apply multi-agent Q-Learning to obtain the optimal solution. 



§ EXPERIMENTS

We consider two microgrids operating on the solar and the wind power batteries respectively. Demand values are taken to be 8,10 and 12. The probability transition matrix for the Demand is given below. 


    P =   [[ 0.1 0.6 0.3; 0.3 0.1 0.6; 0.6 0.3 0.1;     ]].


The maximum storage capacity of the batteries is set to 5. The renewable energy generation process for simulation purposes is taken to be Poisson with mean 2. Discount factor of the Q-learning algorithm is set to 0.9.

We begin our simulations with the state [8,5,5]. First we run Algorithm 1 for 10^6 iterations and compute the average deficit in demand and supply. Then we run Algorithm 2 for 10^8 iterations with ϵ-greedy policy with ϵ = 0.85 and obtain an optimal policy. We then compute the average deficit using optimal Q-values for 10^4 iterations. We compare both the algorithms in Figure 1.




With regard to the problem 2, we plot the values of average deficit in the power and the average production at the main grid obtained by different values of c in (<ref>). The maximum power allocation at the main site is taken to be 8. This is shown in Figure 2. 





 §.§ Observations




  * In Figure 1, we see that the Algorithm 1 performs better than the Algorithm 2, when the Maximum Power Allocation (MPA) at the main site is less than 3. This is because, when the MPA is very less compared to the minimum demand value, storing the power in the batteries doesn't provide any advantage. 

  * On the other hand, if the MPA is not very small and comparable to the minimum demand, the Algorithm 2 outperforms Algorithm 1. Thus, in this case it is useful to store the power in the batteries. This is the main result of our paper. 

  * In Figure 2, we observe that as the value of c increases, the average demand-supply deficit decreases and the average power obtained from the main grid increases. 




§ CONCLUSION

We considered the problem of electrifying a village by setting up microgrids close to the village. These microgrids have access to the renewable energy storage batteries and also electrical connections from the main grid. We identified two problems associated with the microgrid. First problem is to minimize the expected long-run discounted demand-supply deficit. We model this problem in the framework of MDP <cit.>. This formulation doesn't take into consideration the cost of power production at the main grid site. Finally, we formulated MDP taking the cost of power production into consideration. We applied Multi-Agent Q-Learning algorithm to solve these problems. Simulations show that, when maximum power allocation at the main site is not very less, storing the power in the batteries and using them intelligently is the better solution compared to not using the storage batteries. 



§ FUTURE WORK

As future work, we would like to consider the possibility of power sharing between the microgrids. In this case, along with decision on amount of power to be used from stored batteries, microgrids also have to make decision on the amount of power that can be shared with others. We would also like to consider the  heterogeneous power price system. In this scenario, the price of power production at the main grid will vary from time to time. 



§ ACKNOWLEDGMENT


The authors would like to thank Robert Bosch Centre for Cyber-Physical Systems, IISc, Bangalore, India for supporting part of this work.


 IEEEtran
 





