Continuous control with DDPG: How to eliminate steady state error? Currently I'm working on a continuous control problem using DDPG as my RL algorithm. All in all, things are working out quite well, but the algorithm does not show any tendencies to eliminate the steady state control deviation towards the far end of the episode.

In the graphs you can see what happens: 

In the first graph we see the setpoint in yellow and the controlled continuous parameter in purple. In the beginning, the algorithm brings the controlled parameter close to the setpoint fast, but then it ceases its further efforts and does not try to eliminate the remaining steady state error. This control deviation even increases over time. 



In the second graph, the actual reward is depicted in yellow. (Just ignore the other colors.) I use the normalized control deviation to calculate the reward: $r = \frac{\frac{|dev|}{k}}{1+\frac{|dev|}{k}}$.

This gives me a reward that lies within the interval $]0, 1]$ and has a value of $0.5$ when the deviation $dev$ equals the parameter $k$. (That is the parameter $k$ kind of indicates when half of the work is done)

This reward function is relatively steep for the last fraction of the deviation from $k$ to $0$. So it would definitely be worth the effort for the agent to eliminate the residual deviation. 

However, it looks like the agent is happy with the existing state and the control deviation never gets eliminated. Eventhough the reward is stuck at ~0.85 instead of the maximum achievable 1.


Any ideas how to push the agent into some more effort to eliminate the steady state error? 
(A PID controller would exactly do this by using its I-term. How can I translate this to the RL-algo?)

The state presented to the algo consists of the current deviation and the speed of change (derivatve) of the controlled value. The deviation is not included in the calculation of the reward function, but in the end we wat a flat line with no steady state deviation of course.

Any ideas welcome!

Regards,
Felix
Could you figure out a workaround for this problem of the steady-state error using DDPG?

I'm currently facing the same problem. My application is Satellite attitude control, and no matter which cost function variation I use, the resulting controlled system maintains a constant steady-state error.

It seems to be a limitation of the algorithm. But I can't understand why this happens. Hence, I've decided to use PPO for now. But would like to know if you could solve this problem using DDPG.

All the best,

Wilson