title: DDPG not converging for a simple control problem
body: <p>I am trying to solve a control problem with DDPG. The problem is simple enough so that I can do value function iteration for its discretized version, and thus I have the &quot;perfect&quot; solution to compare my results with. But I want to solve the problem with DDPG, in hope to apply RL to harder versions of it.</p>
  <p><strong>Some details about the problem</strong>:</p>
  <ol>
  <li>The control space is [0,1], the state space has dimension 2</li>
  <li>This is a stochastic environment, transitions between states are not deterministic</li>
  <li>There is some non-constant reward at any period, so sparse rewards should not be a problem</li>
  <li>Value function iteration takes just 10 minutes or so, again, it's quite a simple control problem</li>
  </ol>
  <p><strong>What is the issue</strong>:</p>
  <p>My agent always eventually converges to a degenerate policy with action being either always 1 or 0. At some point while training it can be a bit close to the right policy, but it's never really close.</p>
  <p>Similarly, I usually fail to get right the shape of Q-function:</p>
  <p><a href="https://i.stack.imgur.com/4T7rlm.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/4T7rlm.png" alt="Q-value for a typical state as a function of action" /></a></p>
  <p><strong>What I tried to do</strong>:</p>
  <ol>
  <li>I put a sigmoid layer at the end of the actor network (naturally, as the control space is [0,1])</li>
  <li>I have target networks both for policy and critic</li>
  <li>I clap my actions so they are between [0.01, 0.99] (the perfect solution is always within these boundaries)</li>
  <li>I tried adding some artificial penalty to reward for actions close 0 and 1. Then, the algorithm converged to something else, but again not to something good.</li>
  <li>I tried both random uniform exploration or adding small normal noise. I either decrease exploration rate with time or hold it constant but small</li>
  <li>To check my code, I ran the following experiment. I would first fix policy to be the &quot;perfect&quot; one and update only critic. In this case, I manage to learn Q-network pretty well (the shape too). Then, I freeze the critic and update only actor with the DDPG updating rule. I manage to get pretty close to the perfect policy. But when I start to update actor and critic simultaneously, they again diverge to something degenerate.</li>
  <li>I experimented a lot with my hyperparameters, currently they are the following:</li>
  </ol>
  <pre><code>&gt; Optimizer: Adam. Learning rates: 0.001 for actor, 0.01 for critic. Batch size = 50,
  &gt; memory size = 10,000.  Standard deviation of normal exploration noise = 0.02.
  &gt; Weight for soft updates of target networks: 0.01. 
  &gt; Sizes of hidden layers for actor and critic: [8,16,16,8].
  &gt; Length of simulation: from 1,000 to 1,000,000.
  </code></pre>
  <p>I would be very grateful for any advice, thanks!</p>